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SCHOOL DISTRICT EVALUAT I ON OFF I CE5T:~^ 
. ARE THEY WORTH THE MONEY? 
by 

Richard C. Williams and Pamela McGranahan 

Introduction 

It now seems clear that the era of federal support of school district 
education programs is drawing .to a rapid close. Such programs as hot 
lunches, bilingual programs, and ESEA Title IV, to name a few, are either 
being cut back drastically or being eliminated altogether. At the same 
time, federal policy is moving toward block grants, which means that money 
will be funnel ed through the state educational agencies to the school dis- 
v tricts; and school districts will have considerable discretion in determin- 
ing how and when such funds will be spent. 

These basic changes in federal support and policy will have serious 
ramifications for school districts and their clients. School districts 
will likely be faced with considerable turmoil as^ they struggle to sort out 
the claims various constituent groups will make on an ever shrinking budget 
of discretionary funds. 

While attention has been focused on some of the obvious implications 
of this change, the fate of school district evaluation offices has received 
little attention. In most districts these evaluation units have either 
developed or grown directly as a result of federal and state educational 
policy. Most such programs carried a provision that continued funding 
would depend in part on the districts providing evaluation reports that 
showed evidence that the programs were being administered according to 
established policy and that the programs were achieving desired goals. 
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While some districts were ab'ie to meet these evaluation requirements by- 
hiring external evaluators, many districts eventually "pooled" the. evalua- 
tion funds from the many projects into a ui strict evaluation unit. In some 
instances these units were combined with already extant district testing 
offices. Parallel to school districts 1 developing these offices was the 
development of evaluation as a separate field of research and inquiry. A 
federally-funded Center for the Study of Evaluation was established at 
UCLA, many universities developed evaluator training programs, special 
evaluation journals and associations were started. Evaluation became a 
commonplace phenomenon in many school districts and an established field of 

, study. 
The Problem 

The questi ons that are, r or will-be -facing many-school -distri ct 

administrators and boards are, "What shall we do with the district 
evaluation unit now that block grant funding no longer mandates specific 
evaluation? Should we use scarce resources to continue an evaluation 
office. Do the benefits of such offices justify the costs?" 

There is considerable evidence that in many school districts, evalua- 
tion units have never played a very significant role in local school dis- 
trict decision making, in spite of their potential to do so. CSE-sponsored 
survey research and case studies of school district evaluation offices 
suggest that in many districts, evaluation offices have mainly collected 
and reported data to external funding agencies. In other districts, evalu- 
ation offices have disseminated much data, particularly testing data, with- 
in the district; rarely has anyone in the ; district, however, correlated the 
data with instructional activities, so that data-based instructional 
changes are made. 

6 
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Does, this mean most districts should drop or cut back their evaluation 
offices? Not necessarily! A decision to cut back evaluation units, while 
perhaps justified in some instances, might be shortsighted for three 
reasons: 

1) the evaluation unit's past activities and perceived limited impact 
may not accurately predict the evaluation unit's decision-informing 
potential. Indeed, there may be understandable reasons why the 
present evaluation unit has been limited in its scope, such as lack 
of funding or overwhelming dictates from external funding agencies 
that have severely curtailed the unit's choice of activities or 
direction. 

2) There is evidence that the field of evaluation ir- developing in 
such a way that evaluation research can become a genuinely valuable 
decision-making tool for local school district boards, adminis- 
trators, and teachers. 

3) There seems to be increased administrative understanding of how 
evaluative information can be used to serve district needs. 
Administrators' are 'using evaluation information in such diverse 
ways as: justification of budget requests; explanations to the 
public and parents about what is going on; input into decisions 
aoout text adoptions; staff development; and local school planning. 

O f 

There are several reasons why evaluation may be more useful to school 
districts in the future than it has been in the past. One is essentially 
technical -- that is, the state of the art and the cost of computers and 
related software has undergone a virtual revolution. School districts can 
- now have available at an increasingly reasonable cost computer terminals at 
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local school sites. These terminals provide educators, teachers and prin- 
cipals with an enormous tool for having readily available data, relevant to 
administrative and institutional decision making. Jhe evaluation unit can 
play i% vital role in gathering, analyzing and displaying data for school 
site and district decision makers. 

A related development has been the maturing of the evaluation field 
itself. In its early years, the field was largely preoccupied with evalu- 
ation design and related methodologies. After considerable progress had 
been made on that front, evaluation specialists began increasingly turning 
their attention to the question of evaluation utilization. That is, they 
realized that even the best designed evaluations are .worthless if no one 

t uses them. Recent attention has been directed at developing evaluations 
and designing district evaluation-activities-and-processes in such a way 
that they can be of maximum use to district decision makers at all levels. 
What is more, a number of school districts have designed evaluation acti- 
vities and procedures that have resulted in the evalution units being 
directly integrated into school district decision-making activities. In 
such, districts, the evaluation units have begun to fulfill a management 
information system potential. 
A Basis for Decision 

We suggest that as school districts begin the process of deciding what 

. to do about these evaluation offices, they not make such decisions hastily 
or simply on the basis of past experience. Instead we suggest that as 
districts face this decision regarding their evaluation unit, they begin a 
multi-level inquiry into the unit's past performance and future potential. 

8 
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Such an inquiry might include: 

1) Reviewing the unit's past work 

The evaluation' unit, or an independent agency, could present a re- 
view of the work the unit has completed over some period of time, 
such as the past five years, asking such questions as: what data 
have they collected and analyzed; what evaluations have they com- 
pleted: what, in the evaluation unit's view, has been the impact of 
their work; what evidence is there that the work has contributed to 
informing decisions; why was the' work done — to satisfy external 
requirements, in-district requests, or at the office's instigation? 
A part of this review should consist of a survey of district staff 
(e.g., central administrators, site administrators, and teachers) 
asking their perceptions of the evaluation unit. How efficient and 
effective has the office been? What use, if any, have these vari- 
ous constituent groups made of the evaluation unit's work and 
reports? 

2) Identification of work the unit and its clients want the unit to 
perform 

^aluaS^^^ce personnel TouTdT describe the work "and Fthe" kinds'" 
of evaluation reports they would most like to develop, including 
estimates of district capacities (such as money, expertise, compu- m 
ters) for doing the work. This could include a skills analysis of 
the people in the office, describing their strengths and knowledge 
in the areas of data collection, analysis, presentation, and 
interpersonal skills. Also, the district staff could contact other 
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districts and learn about the kinds of functions and services being 
provided. 

The evaluation office's various constitutent groups could identify 
the work and kinds of reports they would really like to be 
available and whyT 

3) Establishing joint priorities 

Representatives of the evaluation office and the various consti- 
tuent groups could meet together to analyze what the office has 
been doing, and what it could be doing differently. Together, the 
group could clarify perceptions, establish priorities, and develop 
a list of potential, high priority activities. 

4) Developing a plan of action 

The evaluation unit could be asked to develop a plan to describe 
potential implementation of the previously agreed upon activities 
and products. An integral part of the development plan should be 
descriptions of ways that utilization of data generated from the 
office will be fostered. At this point it would be useful to the 
office, if necessary, to explore programs and procedures that have 
been developed in other districts to achieve similar goals; Such 
activities would probably have to be modified to meet specific 
local conditions, but there is no reason to start from scratch in 
such activities as: developing a criterion-referenced testing pro- 
gram, developing an instructional continuum, developing evaluation 
reporting formats that are easily read and understood by local 
school site educators or parent advisory committees, or in 

10 
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developing inservlce training programs that will increase staff 
understanding and use of evaluations and test results* 
If this* process was used, appropriate district decision makers would 
have some basis upon which to make a decision about the future directions 
of the evaluation office that is more logically and fully developed than by 
simply extrapolating from past experiences. Such decisions are jTtfficul t 
in these times of shrinking budgets. But distvMct^evalTfation offices can 

play an important role in district administrative and instructional , j 

i 

management. We urge* districts to take a careful and fully informed look at' 
their units, and explore their potential use to the district. / 
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HOW TWO EVALUATION OFFICES HELP 
IMPROVE SCHOOL PERFORMANCE 

by 

Larry Crabbe, Evaluation Specialist 
San Juan Unified School District, California 

Theron Swainston, Associate Superintendent 
Clark County School District, Nevada 

Riqhard C. Williams, Associate Professor 
UCLA Graduate School of Education 

■ ■ ' ' - 



Introduction ^ ' 

The^afgument presented in the preceding article ([reference to journal 
article]) ctn easily be dismissed as some academicians' ideological 
dreams. After al 1 , school districts have managed, for decades, to survive 
without extensively utilizing evaluation offices. B^ut' increasingly, the 
public and. school staffs want to do more 'than merely Survive. They want to 
develop really effective instructional programs. We believe that testing 
and evaluation, when considered as part of an instructional management 
information system, can play an important role in a district's instruc- 
tional improvement efforts. — . ■ 
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Below, we describe two school districts' instructional improvement 
porgrams in Clark County, Nevada and. San Ouan, Cal ifornia, and discuss the 
key role testing and evaluation units have played in these districts' 
efforts. | I 

Clark County School District, Las Vegas, Nevada 

In .1981, the Clark County School District, Las Vegas, Nevada, was the 
• nation's twenty-second largest district; from 1971 to 1981 it showed the 
highest population growth \rate of ^the^fefty largest districts. In order 
for board Members and administrators to supervise and operate such a large 
district's instructional program, they must have extensive knowledge of 
what is happening in schools in relation to what should happen . It is in 
this context that the^valuation office (Research and Development Depart- 
ment) in Clark County has had special meaning and value. 

In the late 1960's, the district evaluation staff's role was quite 
peripheral, probably typical of what one would fTnd in many school dis- 
tricts. They spent most of their time and resources generating data and 
evaluation reports about federally subsidized programs. These reports, 
while qenerally well done, were intended largely for external funding 
agencies; they were seldom used for internal district instructional 
improvement. ■ — 
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In the early 1970's some district administrators realized that the 
instructional program had become deficient. There was very little consis- 
tency in curriculum and instructional practice from school to school ; the 
district's standardized test scores were nearly all below the 50th per- 
centile, with many in the 20's. 

13 
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Some urged the district to begin developing a comprehensive school 
instructional management and accountability system. The resulting plan was 
built on three basic components: 

1) specific instructional and curricular goals and objectives must be 
clearly written and widely communicated; 

2) methods for measuring whether or not instructional objectives were 
attained must be devised; 

3) program improvement decisions should be based in part on assessment 
data. 

The Research and Development Staff played an important role in design- 
ing and implementing these components. 

First, the district established a set of instructional objectives, 
with accompanying evaluative criteria ("What should be"). These ranged in 
scope from simple math skills in the Kindergarten Curriculum Guide to 
annual priority goals adopted by the Board. At the school level, the heart 
of this component was a set of objectives and standards for school opera- 
tion known as Elements of Quality. 

The second component consisted of both formative and summative mea- 
sures of the extent to which stated objectives were achieved ("What is"). 
The evaluation staff administered norm-referenced (nationally standardized) 
tests in grades 3, 6, 8, and 11 and was integral in developing criterion- 
referenced tests (CRT's) to measure basic skill subjects in grades 1-8. 
Three forms were developed: 

. 0 one CRT was a general placement test administered at the beginning 
of the school year; 

0 another CRT was a series of diagnostic instruments used to identify 
students 1 specific learning needs; . 

0 another CRT was. an end-of-year measure of mastery of specific 
instructional objectives. 

14 
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The evaluation staff also developed structured surveys of parent, 
student, teacher, and principal opinions and devised methods for directly 
observing instructional practices and student behavior. 

The system's third component addressed the ultimate challenge of any 
data-based instructional improvement, namely, how to reduce the difference 
between "What should be" and "What is". 

Utilizing testing and evaluation data, teachers, administrators, and 
the school board compared "What is ,! with "What should be," snd decided upon 
high priority and realistically attainable objectives and activities which 
offered the greatest likelihood for instructional improvement. These 
included: developing a teacher inservice training program, improving the 
principals 1 clinical supervision skills, and devising a supervisory system 
that insured that instructional improvement was a top priority at all 
district levels. 

Since implementing this system, measured student acheivement in basic 
reading and math through the eighth grade has increased approximately 20 
percentile points. While exact causality is difficult to determine, we 
believe the program has contributed greatly to this measured growth in 
pupil achievement. 

The district's teachers, administrators— and-board-members-now have a 
better control of our instructional' destiny. In addition to a carefully 
developed instructional improvement program, we have a data-base of 
systematically collected and analyzed test results and evaluation reports 
that help us in Identifying problems and determining solutions. This 
program could not exist without a capable evaluation staff that has and 
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continues to provide leadership and assistance at all stages of the 
process. 

San Juan Unified School District, California 

In 1971, the San Ouan Unified School District was like most school 
districts in the way it handled the evaluation of its ESEA Title I 
programs. The development of a single annual program plan, and year-end 
evaluation report was the focus of its attention. These documents were - 
faithfully submitted to the appropriate governmental agencies. However, 
the value of these evaluation activities to those at the site level was 
exceedingly limited. 

A number of us, including many at the site level, believed that a 
greater return would be obtained if planning were shifted to the school 
level. In 1972, we received an additional incentive. The Early Childhood 
Education program became law and with it a mandate for site-Oevel planning 
and evaluation. The law also created site advisory committees to carry out 
these activities. 

The district began developing a site centered ev N aluati on/planning 
model to help make school level planning a reality. The special projects 
evaluation unit (consisting of one specialist) assumed the .responsibil ity 
for its development. 

The first task was to define a school site planning process. The 
result was a six-step school site planning procedure: 
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(1) 


recognizing the problems that exist; 


(2) 


determining their causes; 




seiecxing ai xernavive solutions, 


(4) 


selecting the best al ternative(s) ; 


(5) 


implementing the selected al ternative(s) ; 


(6) 


measuring its impact or success. 



Most critical to~the success of our model was the belief that 
evaluation and planning must be closely linked. Several additional factors 
were (and are) considered to be important to the success of ths model as 

wel 1 : I 

I 

0 it must provide data that are important to the planner — not only 
data of Interest to evaluators; 

M ' ■ 

0 the reporting format must be such that the meaning of the data is 
clarified rather than confused; 

0 the report must fit into the planner's decision-making time cycle 
— well in advance of the decision deadlines. 

An important district concern had to do with who was to participate in 
the school site evaluation and planning process. In order to maximize 
ownership and commitment the school planning councils were organized to 
include parents, teachers, and (in secondary schools) students. 

Working with school site council members, the district identified the 
data needed to make enlightened site decision making possible. Some of the 
data related to achievement as measured by norm-referenced tests; other 
data had to do with the opinions of significant groups (i.e*, staff, 
parents, and students) toward their school's program and progress. To 
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provide these data, the district developed the instruments of the 
Educational Program Assessment model. Its surveys contain items common to 
all schools as well as an opportunity for each school to add questions 
tailored to its needs. Data from the EPA instruments, norm-referenced 
tests, and a variety of other sources provide a rich base upon which to 
plan. 

School site councils can, however, drown in floods of data. What is 
more, even if such data are understood, their implications For planning 
purposes are often murky. Several steps were taken to address this 
problem. First, we devised a procedure that reduced 300 test printouts 
down to six school profiles that graphically point out school performance 
and problem areas. These profiles include the information for six years, 
five grades, and s subtest areas. We arranged the data in order to 
facilitate comparisons between the various grades and years. The results 
of the EPA instruments were also produced in a concise graphic format. 

To make site decision making more manageable, we divided the members 
of each school site committee into small groups t each completing the six 
steps of the planning process for a particular area of the curriculum. 

Finalli we entered each completed school site plan and budget into a 
district word-processing system to enable school sites to modify their 
plans and monitor their budgetary status throughout the school year. This 
has greatly reduced. the amount of paperwork at each school site: 

As a result of these efforts, school advisory councils are no longer 
merely "rubber stamps." They now have the data and the means by which to 
make informed planning decisions and to assess, from year to year, whether 
or not the resulting program has produced the desired results. 



It has taken the district about eight years to devise this system. 
(We have, unfortunately, not been able to discuss all of its components 
here.) Many parents, teachers and administrators have helped to think 
through its problems and solutions. They have been active participants 
throughout the model's evolution. We are now enjoying the results of ouv 
efforts. School site councils report an ability to play an important role 
in determining the destiny of their schools. They have a way to "stay on 
top" of things and to learn of' the outcomes of previous decisions. Needed 
program .adjustments can readily be made. It is rewarding to view the skill 
and confidence with which local staffs and parents fulfill their planning 
responsibilities. Communication and mutual support between school and 
community have, in many instances, never been better. Most importantly, 
our students' educational achievement appears to be steadily improving. 

We are not devising a system to utilize individual site data for 
districtwide planning and evaluation purposes. If successful , our site 
level information will also satisfy district needs with no additional data 
collection burden to schools. We can, then, better coordinate school -level 
planning activities with those which are best conducted at. the district 
level. 

It is our belief that the entire planning/evaluation process could not 
take place (especially in large districts) without a well trained 
evaluation staff that views the facilitation of effective and efficient 
site-level planning as its highest calling. 
Conclusion . ' , ' 

In 'comparing these two -districts, one sees important differences and 
similarities. One difference is that curricular and instructional decision 

19 
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making takes place at different locales. While both consider instruction 
an important district repsonsibility, Clark County uses a more centralized 
approach which stresses a set of commonly agreed-upon objectives and 
instructional methods in all schools; San Juan emphasizes more local school 
site decision making which reflects the school site council's 
interpretation of local conditions and needs. Both approaches are 
effective; testing and evaluation systems can serve multiple approaches, . 

The systems are similar in that both have transformed a procedure that 
made minimal use of testing and evaluation data to one that makes such data 
a vital part of the instructional decision making. As "pioneering 11 
districts in their efforts, it took each about eight years to develop, 
perfect, and ; install its program. ^ While these districts 1 exact programs 
wi^ll not likely work equally well in other districts, the programs and 
their component parts can provide a foundation of ideas and procedures that 
can be used by other districts that are interested in utilizing testing and. 
evaluation data as district instructional resources. 
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SCHOOL DISTRICT USE OF TESTING AND EVALUATION 
FOR INSTRUCTIONAL DECISION MAKING: A BEGINNING* 



Adrianne Bank and Richard C. Williams 

During this decade of the '80' s, educational organizations — like 
profit-making and governmental organizations are experiencing, consider- / 
able stress. School systems are being asked to respond to demands for 
increased excellence in educational quality while their financial resources 
are declining. If educators are to secure the funds necessary to meet the 
diverse needs of a heterogeneous, mobile student population and tc raise 
the low status and low salaries of teachers, and administrators they are 
going to have to devise ways to more actively and effectively manage their 
instructional programs and then let the public know about it. An important 
ingredient in this process is information for teachers, principals, and the 
public about the effects of the instructional program and discussion 
concerning the implications for action*. 

School district offices currently obtain a great deal of information 
about student achievement. They also know some details about the operation 
of their instructional programs. Over the recent past the districts 1 human 
and technical capacity to collect data about students and programs has been 
enlarged by federal and local need-to-know reports concerning the outcomes 
of funded programs. Most large districts now have research and development 
offices. Many districts routinely administer standardized, norm- 
referenced, criterion-referenced tests, and proficiency tests. Many dis- 
tricts annually distribute parent, teacher, and student cpinionnai res. The 

*Presented at the symposium: Local level evaluation: The state of 
practice in health, housing, education and urban services. Evaluation 
Network/Evaluation Research Society, paltimore, Maryland, 1982. ; 



use to which this data is put, however, varies widely from district to dis- 
trict. Some districts largely report the data to external funding agen- 
cies; other districts utilize the data for internal district instructional 
decision making. 

For the past three years, the authors have been conducting NIE spon- 
sored field research at UCLA's Center for the Study of Evaluation in order 
to understand what distinguishes districts who make internal, system-wide 
instructional ly-oriented use of student test scores or other evaluative 
data from those districts who use mandated tests scores and evaluation 
reports primarily to comply with federal and state requirements. 

We are aware of the numerous technical difficulties related to devel- 
oping test instruments and analyzing test scores as well as the other eval- 
uative data from surveys, classroom observations, etc., in ways that en- 
courage appropriate inferences >f or instructional policy making. We acknow- 
ledge that even when these technical difficulties are overcome, generic 
organizational characteristics of school districts as organizations make 
the kind of information-based instructional management systems we have seen 
in some districts relatively rare occurrences. 

However, we have discovered some common conditions in the eight dis- 
tricts we have studied that appear to contribute to their ability to forge 
systemic linkages between testing and evaluation data on the one hand and 
instruction on the other. We believe that similar facilitating conditions 
may exist in other districts, or can be brought into existence, if the 
motivation is sufficient. Our study districts are 1n what we have labeled 
"phase three" in regard to data collection, analysis, and use, that is, the 
integration of testing and evaluation with instructional decision making 
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We are not certain at this time if this third phase phenomenon will dis- 
appear or whether it marks a beginning of a useful movement in district 
instructional management. 

BACKGROUND 

In earlier, simpler — phase one ~ times, teachers used their own 
made-up tests to motivate students, to determine levels of student 
achievement in order to provide remedial or enrichment activities for them, 
and to assign required report card grades. Teachers, autonomous in their 
classrooms, were responsible for testing students, for evaluating their 
performance in other ways, and for teaching them. Districts did not 

require system-wide tests. 

During the recent twenty-year history of high levels of federal and 
state involvement in the formulation and funding of educational programs, 
teachers continued to use self -developed tests, but district offices began 
to require additional tests or evaluative record keeping, either on their 
own initiative, or because of requirements accompanying externally funded 
categorical programs. Many districts now routinely require the administra- 
tion of annual or semi-annual standardized tests at three or more grade 
levels, periodic criterion-referenced tests, proficiency tests, and state 
assessment tests. About half of the testing occurring 1n elementary 
schools 1n reading and math, and about one quarter of the. testing occurring 
in high school classrooms 1s Initiated, by someone other than the classroom 
teacher (Burry, et al., 1982). Many observers, (David, 1978; Lyon, et al, 
1978, Zucker, 1981) point out that, for the most part, test results are 
sent out of the district to fulfill reporting requirements. Whatever 
internal use 1s made of them is likely to be by administrators of thej 
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programs, rather than by teachers or principals. It is our finding as well 
as those of others (Kennedy, et al.,1981; Alkin, Daillak & White, 1981) 
that most districts do not relate their district-wide testing and evalua- 
tion activities to their ongoing instructional programs. Those districts 
where district-wide tests are routinely administered or where evaluative 
record keeping is performed but where test, scores, and evaluative data 
are rarely scrutinized for policy or classroom purposes could be said to be 
in phase two with regard to testing and evaluation activities. 

In a small, but grtv i ? , number of districts, school boards and admin- 
istrators have found a utilize for Internal decision making the 
Information collected by their recently acquired research and evaluation 
personnel. These boards and administrators have decided, 1n response to 
their own, to teachers 1 , and to the public's demands for better pupil 
achievement that test scores and other data- can guide policy and classroom 
• decisions as well as track student performance. These districts, due 
perhaps to particular circumstances, have developed what we are calling 
instructional information systems. These systems use Information derived 
from a variety of data sources, e.g., student scores on one or more test 
formats, parent surveys, demographic trends, to analyze, and perhaps 
change, Instructional programs, Including texts, supplementary media or 
materials; supports or training for teachers; amount of time on specific ■ 
subjects, etc. 

This "phase three" linkage of testing, evaluation, and Instruction 1s 
'characterized by district acknowledgement of system-wide responsibility for 
student learning and by district confidence that test scores and other 
evaluation data can be used as planning and assessment tools. 

o 
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The two questions addressed by this paper are: Why have some, but not 
other school districts moved towards the development of system-wide link- 
ages between testing and/or evaluation with instruction? What do existing 
linkages, that is, instructional informatipn systems, look like? 

WHY SOME DISTRICTS AND NOT OTHERS? 
We have identified several generic characteristics of school districts 
as complex organizations that tend to keep most of them at phase two, that 
is, lots of test data collected at district behest, little test data used 
at district direction to improve the instructional program. 

These characteristics predispose district central offices towards 
short-term reactions to their ever-present and constantly changing problems 
rather than towards sustained long-term efforts to improve instruction in a 
cumulative and systemic manner. They also predispose central office re- 
search and testing units towards autonomous, in contrast to coordinated 
functioning. Furthermore, individual teachers or schools assume, impli- 
citly or explicitly, delegated authority for. most classroom decisions. 
Central office responsibility for instruction is limited to inconsequential 
support and supervision activities. These generic characteristics include: 
° rapidly changing environmental conditions . Fluctuations in dis- 
trict budget levels; changes in student population characteristics 
and distribution within the district; reductions in staffing all 
make it difficult for district administrators to find the time, 
resources, or constancy to develop a coordinated instructional 
improvement effort monitored and guided by information from tests 
and evaluations. 



boundary permeability . There is a great deal of external societal 
regulation of districts and schools. Specific laws and Regula- 
tions, as well as changing and sometimes conflicting community 
expectations, may dispose district' administrators : .Jto^ft^end..f1.cst 
and foremost to those tasks for which they are legally arfd politi- 
cally accountable even if those tasks do not most directly con- 
tribute to instructional excellence (Zucker, 1981). 
goal diffuseness . The pluralism and diversity within our society 
and within any given district may make it difficult for districts 
to adopt educational goals specific enough to tune tests, texts, 
instruction to them. In order to avoid offense, educational goals 
are often stated in ambiguous language. 

weak technical core . Some observers have noted that education's 
weak technical core — that is, the small number of credible 
research-validated cause-effect relationships between student 
instructional methods and learning outcomes — encourages educators 
to use and justify a variety of instructional methods. Arguments 
have been made (Meyer & Rowan, 1977) that revelation of this weak 
) technical core may be detrimental to a district's public image and 
; therefore administrators leave such decisions to teachers who have 
the imprimatur of being credentialed. 

loose coupling . A widely held view of the internal operations ^of 
public school districts is that the instructional components of the 
organization, e.g., curriculum department, in-service training, 
individual classrooms, are loosely coupled (Meyer & Rowan, 1977; 
Weick, 1976). It follows that the linkage between policy and 
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management and between management and operations is likely to be 
weak. Loose coupling within a district jnay be-more-pronounced in — 
the complex areas, such as instruction, which are protected from 
public view than in simpler, but more visible areas like the 
enforcement of policies relating to corporal punishment. m Instruc- 
tional^, schools and classrooms may be thought of as federations 
or zones of Influences (Lortie, 1975), rather than as places where 
board or central office policies necessarily are carried out by 
teachers in classrooms. 
In light of these generic conditions, what are the situation-specific 
elements present in those districts which have developed information-based 
instructional management systems? The elements can be conceptualized as 
the interaction between a benign external environment (Pincus & Williams, 
1977) and the organization's capacity to respond. 

! In the districts we studied the thrusts from their many environments, 
for example, federal, state, local, media, academia, parents, were aligned 
in the same direction. Conjointly, these pressures pushed the district 
towards making instructional changes to increase student achievement as 
reflected in test scores. In addition, it seemed to us that the political, 
social, and economic situations in our study districts were somewhat less 
turbulent, somewhat more^stable than in other districts where crises seemed 
continually chronic. j 

Out study^dfstrictrwere able to respond to the direction suj&^wd by 
the pressures from /the environment because of: the presence of influential 
"idea champions," and of a stable core of central office personnel, staff 
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orientation towards comprehensive problem analysis, and an administrative 
capacity for dealing with ambiguity and delay (Bank & Williams, 1981). 

Idea champions were defined as individuals in key administrative and 
policy positions who firmly believed in some variation of the following 
position: that classroom instruction could be conceptualized in terms of 
student learning outcomes, that tests of student learning outcomes and 
evaluations of instructional activities could be used to relate, instruc- 
tion to learning, and that district policies and procedures should be 
attentive to this linkage. 

Stable core staff meant that a critical mass of teacher and adminis- 
trative supporters of these ideas had been around for a /while. In dis- 
tricts with the most advanced information systems it had taken more than 
five years for the linkages to develop and mature. A stable core group, 
with some additions or deletions, seemed to have worked' steadily at the 
process over several years. They were thus able to smooth out the shifts 
in direction that might have resulted from changes in school board 
composition, legislation, court orders, funding levels, etc. 

Comprehensive rather than a d hoc problem analysis on the part of dis- 
trict managers meant that the core central . off ice staff prepared for next 
steps beyond the immediate task of the moment. Typically our districts did 
not davelop elaborate, multi-year blueprints of their projected information 
management activities in advance of action. However, from the beginning 
they did have a sense of where they were going and the steps that were 
necessary to get there. 

District personnel In our study -seemed to have the ability to deal 
with delay and ambiguity; that is, they were aware of, and accommodated to 
a strategy for developing an instructional information system that was * 
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uneven, interrupted, and in some cases oft-postponed. Creating arrange- 
"nSTentrTonire into .integra- 

ted configurations took time, patience, administrative talents, and adher- 
ence to an overall idea rather than to specific details. 

WHAT DO INSTRUCTIONAL INFORMATION SYSTEMS LOOK LIKE? 

The information systems of the districts we studied varied on a number 
of dimensions; among them, the purpose of the system, the number of ele- 
ments which were coordinated, and the locus of decision making. 

Here are thumb-nail sketches of two districts which differ substan- 
tially on these three dimensions. District A. is small and has a central- 
ized, highly coordinated management system, characterized by a district- 
wide curriculum scope and sequence, district-wide-criterion-referenced 
testing, district-wide staff development activities, clear expectations for 
principal supervision, classroom supports in the form of learning and media 
specialists. Its purpose is to provide ongoing information and resources 
' to classroom teachers who>then differentiate their instruction to meet in- 
dividual needs. District B is medium* sized, has a decentralized, school- 
level decision making focus. The district sends out a variety of testing 
and survey data to help school planning teams do their own resource alloca- 
tion. The district's purpose is to provide information and support to site 
teams who then help solve school level problems. The sketches are adapted 
from district administrators 1 own accounts. 

District A: A Centralized Information System to Monitor and Improve 
Classroom Instruction 

In 1970, the district attempted to meet the need for student progress 
reporting by purchasing test items to measure a sequence of behavioral 
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objectives. The objectives themselves turned out to be sequenced dif- 
ferently than those existing in classroom materials. Teachers hated the 
objectives and the tests and eventually district administration discarded 
both. » 

Rather than purchase another available testing system, district admin- 
istration formed teacher committees to write their own tests. The feeling 
was that tests written by district teachers would be better received by 
other teachers than' tests developed by an outside agency. In 1973, the 
first district-developed tests, known as PAL (pupil Assessed Learning), 
were ready and administered to students. They were intended to monitor 
student learning so as to feed back information to teachers about how 
individualized instruction was working. They also were seen as a way of 
reassuring parents who were concerned about their children's progress. 

Teacher reaction to PAL was extremely negative. Everyone complained. 
Teachers today remember their complaints: e.g., "It was a waste of time." 
"It didn't tell us anything we needed to know." "It was not coordinated 
with anything we taught." Teachers loudly made their feelings known to 
parents and to the Board. It was a hard time for central office staff who 
wanted the testing system to work. 

The person in charge of developing the system next appointed teacher 
committees to try to revise the items. Simultaneously, work began on a 
district, level instructional continuum.. A group of volunteer teachers were 
paid to work on a reading continuum during the summer. \ Their work con- 
tinued through the school year. 

By 1976, teacher committees had generated a reading continuum (RIC) 
and the beginning of a math continuum (MIC). By 1976, also, the PAL 
criterion-referenced testing system had been scrapped. Other teacher 
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committees had written reading, math, and language arts test items to form 
their own criterion-referenced tests (CRT). 

Realizing that a testing program by itself could not improve instruc- 
tional practices, the central office instituted a staff development program 
in the early 1970' s. Nearly all teachers are trained in a diagnostic- 
prescriptive approach to teaching. A modified version is provided for sub- 
stitutes and aides. Other staff development programs discuss individualiz- 
ing instruction, meeting affective needs, using inquiry skills techniques, 
and teaching ^specific curriculum content such as reading, writing, or 
mathematics. 

At the same time as the staff development program was being opera- 
tionalizedi the district office decided to create a new school-based role 
— that of a learning specialist whose responsibility was coordinating the 
newly developed testing system in each of the schools. 

The district claims that their testing system holds the curriculum, 
the texts, the staff development, the teacher supervision together. The 
R&D person notes: . 

° All tests are directly linked to a kindergarten through eighth 
grade instructional continuum for reading, language arts, and 
mathematics. 

° All tests are computer-scored. Results afe formatted in easy-to- 
read specially tailored reports to teachers, parents, and site and 
district administrators. 

° Results are reviewed and appropriate actions to remediate children 
not performing well are immediately undertaken and conrntunicated to 
everyone concerned. 

° The tests are reviewed on an ongoing basis with revisions occurring 
* four times over the last several years. 




District B: A Decentralized Information System to Improve Schools 

"Our evaluation/planning model is one which takes place where the 

action is, at the school-site level. It involves those with high 

stakes in its outcome, namely, the school's staff, students and 

parents. Significantly,, our model is one which begins with a broad 

data base. Most importantly, it goes on serving throughout the school 

year, long after the formal process of planning has been concluded." 

Throughout the evolution of their evaluation/planning model, this 

district has held to several fundamental beliefs as to the reasons for 

evaluation and planning. They believe that an evaluation and planning J 

model would deliver the greatest "payoff" to the district's schools 1f 

these are Its primary goals: 

•° to improve the quality of the local educational program; 

° to encourage the most effective or efficient use of scarce 
" financial resources; ■ .„ 

° to assist with the attainment of the goals of the school, the 
school district, and the special program(s) mandating the 
evaluation/planning activities. 

Ten years ago, when the Evaluation Specialist position was created, 
the district had six elementary schools in Its Title I program. School 
personnel continually asked that they be allowed to create their own pro- 
gram plans suited to -the uniquenesses of their Individual school sites. 
Serious doubt was often expressed by federa) or state administrators with 
regard to the ability of local school sites "to accomplish such a task. 

However, 1n 1971, the legislature passed a bill .authorizing the Early' 1 
Childhood Education (ECE) program and, with 1t, a framework for evaluation 
and planning at the school level, the California State Department of 
Education developed the evaluation/planning model as the planning format. 
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The creation of the ECE.. program prompted the district itself to develop its 
own participatory evaluation/planning model to be based on a broad needs 
assessment process. Their efforts were shaped by the state's process and 
•forms. 

The earliest versions of the staff, parent, and student needs assess- 
ment survey instruments came to be widely known as the Educational Program 
Assessment (EPA) process. Since Its creation, the ongoing work on the EPA 
Instruments Involved large numbers of parents, staff, administrators, and 
secondary students. This has provided district staff with a unique oppor- 
tunity to test out their theories regarding the beneficial effects of - 
Involvement on school and on community feelings of ownership and support 
for such processes. 

"We believe that broad school and community' involvement has contri- 
buted to responsive instruments. These, in turn, have led to high 
response rates, averaging about 70 percent and rising to an unbeliev- 
able high of 100. percent response from one school and Its community." 
While surveys of needs assessment responses provide a large amount of 
Information for' evaluation and planning, the primary source of the school 
district's evidence about student achievement 1s the Iowa Test of Basic 
Skills. 

The evaluation/planning process takes people through a six-step prob- 
lem solving sequence at the school -community level;. Evaluation and plan- 
ning activities take place under the patronage of the School Site Council, 
a body whose composition and Influence has grown out of the California 
School Improvement Program (SIP) legislation. This committee, with the 
site administrator, represents each of the major constituencies within 
school-community (I.e., parents, staff, administrators, and secondary 
students). 
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For planning purposes, the school's curriculum is typically (although 
not always) separated into academ^coinponenfsr such as reading, language, 
mathematics, etc. The School Site Council usually creates "component com- 
mittees" of parents, staff members, and students (in secondary schools) to 
do evaluation and planning for each academic area. The final school pro- 
gram plan has sections addressing each component written by the separate 
committees. 

At the outset, each component committee tries to define its arei of 
the curriculum comprehensively, to ensure that all sub-areas are addressed 
in the evaluation/planning process. The members of the component commit- 
tees examine their information to identify the strengths and weaknesses of 
their students. With this step complete, the component committees turn 
their attention to the existing programs. In those areas where student 
performance has been found to be unsatisfactory, an effort is made to iden- 
tify probable program" causes, that is, program gaps or weaknesses which, if 
eliminated, would result in improved student performance. 

Then, the School Site Council does a component-by-component review of 
all aspects of the^proposed program after an anticipated projected, cost has 
been attached to each proposed expenditure. Through negotiation and com- 
promise, the activities of lowest priority, along with their related costs, 
will- be reduced or eliminated to the point where the amount which the 
school proposes to spend will exactly equal the amount which it expects to 
receive. 

HOW DO INSTRUCTIONAL INFORMATION SYSTEMS VARY? 
Purpose of System . In the two districts described above, the purpose 
of one Instructional Information system was to tailor classroom Instruction • 
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to individual student needs, of the other to improve school functioning. 
Some of our other study districT^l^veliiore narrowly definecTThe - purpose of 
their instructional information system as to raise the average of the dis- 
trict's test scores. Other districts defined the purpose of their instruc- 
tional information system as monitoring teacher behavior. Their idea was 
to assert, district responsibility for instruction by mandating a particular 
instructional strategy and then tracking the effects of that strategy by 
test score analysis. 

Extent of Administrative Coordination . Our eight districts differed 
from one another in the complexity of their coordination arrangements at 
the central office level. Usually autonomous district functions which were 
linked in formal or informal arrangements included: staff development, 
. instructional materials (texts and adjunct materials), testing arid evalua- 
tion, supervision of principals, curriculum development. Coordination was 
carried out by some or all of the following: informal personal Hnks among * 
a few people; formal, inter-organizational reporting arrangements among 
units or departments; on an ad hoc or routine basis, central office staff 
and principals' discussion of district-vide and school site impl icaticns t of 
data analysis. 
Locus of Decision-making 

The foregoing discussion of variations in purpose and in coordinating 
arrangements alluded to a number of organizational levels at which deci- 
sions were made about translating the analyses of test^ scores into changes 
in instructional activities. Districts differed from one another in the 
nature of the deci$ions which were made at the central office level, at the 

—sch o o l 1 eve lr-or— th e classroom-level. — The -f al 1 owl ng .1 j st. 1 ndi cates the 

. range of activities we found at each level. 

- ,,. V ' :■■'/■.: + :■;■. ■>V->.i/^:;-:i; : .^>y?^ i: ^^;- , 
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CENTRAL OFFICE decision-making consisted of one or more of the 
following: w 

° district-level construction of grade-level objectives by 
subject areas 

° district-level construction of criterion-referenced tests 

° district-wide selection and use of a norm-referenced test 

° district-wide evaluation of instruction through classroom 
visits, surveys, etc., of instructional efforts 

° district-wide formulation and conduct of staff development 
programs 

° district-wide selection of texts to match tests 

SCHOOL LEVEL decision-making consisted of one or more of the 
following: 

° school -level planning teams, teacher/parent/community 

° school-level receipt of information about student outcomes 
from either teacher-created tests, teacher-option CRT's, 
district-wide mandated CRT's; district-wide norm- referenced 
tests 

° school-level conduct of instructional evaluation via 
supervision, teacher self-reports, district evaluations, 
outside evaluations 

° school -level planning for school -year activities 

° school-level allocation of services, both.personal and 
financial, to support local plans 

° school -level decisions about texts 

CLASSROOM LEVEL decision-making consisted of one or more of the 
following: 

° individual teacher receipt of Information about student 
learning, e*?., norm-referenced test scores, CRT's, teacher 
observations^ text tests, student assignments, etc. 

° Individual teacher decisions about grouping, remediation, 
enrichment, al ternative instruction, etc. 

° Individual teacher participation 1n professional development 
activities ■ o . 
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CONCLUSIONS 

^ e ~ha Ve ~ come ~ to ~ some ~ten^^ — 

and instruction in "phase three" districts. 

1. Building an instructional iTif ormati on management system can be 
done in districts. It is, however, difficult to do, requires a specific 
set of advantageous circumstances, and takes time. 

2. There are many yet unanswered questions about such systems. For 
example, what do they cost? What are their negative as well as positive 
consequences? Are they worth doing? Can the pace of the development 
process be accelerated? 

3. There is no single method or design which districts have used in 
developing their systems. Whether such systems are uniquely configured 
due to the specifics of community priorities, history of the district, per- 
sonal biases of the "idea champions" or, on the other hand, because the 
state-of-the-art of instructional information systems is so underdeveloped, 
we do not yet know. We do know that there is much district interest in the 
use of information to guide instructional decision making; we suggest that 
the topic of instructional information systems be added to the agenda of 
researchers and practitioners interested in practical district-level 
supports for effective instruction. 
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SCHOOL DISTRICTS. IN THE INFORMATION SOCIETY: 
THE EMERGENCE OF INSTRUCTIONAL INFORMATION SYSTEMS 



By" ' ' 

Adrianne Bank and Richard C. Williams 



Even the most casual observer of modern day America will have noticed 
the breathtaking page at which microcomputers are being introduced into our 
work places and homes. This phenomenon marks an important milestone in 
the journey from an. industrial society to the much heralded information 
society. 

This transition to an information society has tremendous implications 
for everyone — at home, in the officte, on vacation, or in school. To 
date, the educational implications of the technology explosion have re- 
ceived considerable attention as various pundits have tried to predict the 



impact the computer will have on classroom instruction, on curriculum^ 
development and, indeed, on the very structure and purpose of schooling 



which the emerging information society might Impact on schools, namely, how 
school districts can begin more effectively to integrate information into 
their instructional decision-making systems. 

During the past three years at UCLA's Center for the Study of 
Evaluation, we have investigated ways _in which a number of school districts 

have tried to link their testing programs and evaluation/research activi- 
ties with instructional decision making. Our work has included case stu- 
dies in eight districts that had a reputation for having forged some kind 

^ of._test1ng-eyaju*^ In.„addltton rwe ^ have re viewed 




itself. 



In this article we want to describe yet another dimension of ways in 
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and synthesized the theoretical and research literature in several related 
— f 1 el ds-such - as- sch^ 

and management information systems. 

Surveys and field work (Lyon et al, 1978; Bank, Williams & Burry, 1981) 
have revealed that most school districts al ready collect considerable 
amounts of data with potential value for instructional decision making. 
However, these data often remain as unused resources because they are not 
analyzed or reported in a way that is useful for Instructional decision 
making at the classroom, school building, or distri/ct level. 

In our eight districts, we have observed attempts to create useful data 
storage, retrieval, and reporting systems. These we have called instruc- 
tional* information systems (IIS) to suggest a partial analogy with manage- 
ment information systems (MIS). As a contribution toward further under- 
standing what instructional information systems might do, we will, 1 in this 
article: 

1. define what we mean by an instructional information system (IIS); 

2. identify and describe the components of a district instructional 
Information sysiufiir 

3. provide a brief description of three different Instructional Infor- 
mation systems that presently operate in districts; 

4. suggest a direction for a school district instructional information 
system that is compatible with Its organizational context and avoids 
some common errors that sometimes plague management information 
systems. * 

Before embarking on these three topics, let us first share what we 
observed across our sample districts as common elements in the development 
and evolution of existing school district instructional Informatibn sys- 
terns. These elements seemed to distinguish these districts from; other 



districts which collected data, e.g., test scores, but did not make 
instructional use of them: 
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° a stable external environmental setting; 

° wi thin-district presence of "idea champions" and a critical mass of 
long-term supporters; 

° the availability of sufficient resources, both fiscal and technical, 
over a relatively long period of time — six to eight years in many 
cases (Williams & Bank., 1981). 

Another striking element which appeared to characterize our eight 
* districts was the ad hoc nature of the development of the system. In none 
of" the districts had there been a blueprint or a timeline for developing an 
instructional information system. Indeed, the term "instructional in- 
formation system" was not in common use in these districts. Rather, what 
we observed were individual activities, sometimes formalized in job de- 
scriptions but more often informal, which served to combine the collection 
and analysis of data with a deli very, and support system for users at the 
classroom, school* central office, and boarjj levels concerned with 
instruction. ' -* • 

Thus, it is important to' note that the term "Instructional information 
system" used in the remainder of the paper is a construct which we have 
used to bound certain types of district-wide activities. This construct 
suggests that what goes on in school districts which link evaluation, 
testing, and instruction may bear a partial resemblance to what are termed 
management information systems in other organizations. ^ 

What is a School D istrict Instructional Information System? 
i ~ 

In its simplest and most obvious form, an instructional information 
system is some method by which information of some kind is transmitted to 
and used by someone or some group in a school district in relation particu- 
larly to the content or delivery of instruction. But we want to put more 
specific limits on such a definition. 
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The topic of information and its use has been a research subject in 
its own right. There are over 20,000 titles under the term "information" 
in the ERIC system (Mansfield, 1983). Human beings are capable of organiz- 
ing vast amounts of information into patterns which influence their deci- 

. t ... 

■sions and their actions. This "body of knowledge that administrators and 
policy makers use spontaneously and routinely in the context of their work 
... - the entire array of beliefs, assumptions, interests and experi- 
ences - " has come to be called working knowledge (Kennedy, 1982). Such 
working knowledge may often Interact with — that 1s, influence and be 
Influenced by — formal social science data. It is, however, this latter 
type of data — intentionally collected and analyzed 1n a prescribed and 
standardized format — which we are calling "information." 

The term "system," Hke the term "Information," 1s 1n widespread use, 
particularly 1n the literature on organizations. When used here, 1t does 
hot describe the school district as a whole, but rather refers to a 
separate subsystem which has its own purposes, organizational structure, 
staff and linkages to that larger environment. 

Instruction 1s a third broad term which has different meanings depend- 
1 ing on context. We use It here to refer specifically to Intended Inter- 
actions within the classroom or school environment that affect student 
learning. The decisions that affect those. Interactions — such as texts, 
number of aides 1n the classroom, amount of time to be spent 1n a subject 
area, teaching methods may be made by boards, committees, principals, 
teams, or Individual teachers. Whoever the decision makers oh decision 
Implementers, and whatever the topic under consideration, if 1t pertains to 
the schools 1 shaping of students 1 lear^i^g, we count it as instructional. 
: y " • i \ . ' 
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ERIC 



- 39 - 

" As indicated earlier, the term instructional information system was 
chosen to suggest a rough analogy to management , information systems. A 
management information system has been defined by Walter J. Kennovan (1970) 
as "an organized method of providing past, present and projection informa- 
tion relating to Internal operations and external intelligence* It sup- 
ports the planning, control and operational functions of an organization by 
furnishing uniform information in a proper time frame to assist the d^cj-^ 7 
slbn-making process." To parallel this definition, instructional informa- 
tion systems in school districts might be characterized -as loosely orga- 
nized methods of providing to those concerned with Instruction past and 
present information relating to student attainment and program evaluation. 
Instructional Information systems support users 1 decision making by fur- 
nishing them with particular and limited types of Information 1n a time 
framfe and format appropriate to their decision-making processes. 
Components of District Instructional Information Systems 

As noted earlier, district instructional Information systems are 
rarely conceptualized as-such by the people w1th1|fechool and district set- 
tings. The five components, which we categorize as "core" components, are 
terms we derived from the literature on management information systems. To 
greater or lesser degrees these components were present 1n all of our 
sample" districts even though they were not always so named by district 
respondents. . . 

The three additional components which we have labeled as contributory 
are not part of the description of most management Information systems. 
They were, however, also present to SQ^re extent 1n all eight of our dis- 
tricts. They were there to. provide users of the data with guidance and * 

J ... . ' ' . ■ - - ■ . ..- • i 
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assistance for making instructional ly-rolated decisions and with support 
for carrying out those decisions 1n the central office, in schools, and in 
classrooms. Central office personnel who had established the systems often 
noted that these extra-system components were necessary to system main- 
tenance. Without them, they said, it would be likely that principals and 
teachers would revert to exclusive reliance on working knowledge. 

Core components of a district instructional .information system: 

1. specified users 

2. specified uses 

3. v specified types of Information inputs/outputs 
4/ specified Information delivery- procedures 

5. specified monitoring of system functioning and of system use 

Contributory components of district instructional Information systems: 

6/ training for users in data-based decision making and„ 
Implementation 

7. availability of resources to support action planning 

8. availability of resources to support Implementation 

The following: Is a comprehensive listing of the specific elements 

Included by all eight districts within each component. Since districts 

varied from one another on the purposes of their Instructional Information 

systems, only a small subset of the elements of each component was relevant 

to a given district. Following this catalogue we will describe three 

models of Instructional Information systems on a case study basis. 

1. SPECIFIED USERS: 

teaGhers; 
principals; 

others in schools, such as media and learning specialists, 

substitutes, aides; 
advisory committee members; 

parents, media, prospective residents, real estate developers; 
central office personnel concerned, for example, with curriculum, 

supervision, staff development, personnel; 
school board members; 
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These users can be thought of as either direct or secondary users of the 
system; either regular or episodic users; either active or passive users. 

2. SPECIFIED USES: 

planning instruction, identification of subject areas in which 

students 1 need additional time or attention; 
placing, grouping, regrouping of students; 
remediating or supplementing students 1 instruction; 
monitoring student progress; 

identifying parent, teacher, student, opinions and attitudes; 
determining the allocation of school level resources; 
identifying school -wide needs;' 
selecting texts; 

establishing school and district image; 

communicating with interested others - e.g., federal or state 
agencies and local organizations; 

3. SPECIFIED TYPES OF INFORMATION INPUTS/OUTPOTS: 

commercial norm-referenced test and subtest scores; 

district-developed criterion-referenced test scores; 

proficiency test scores; 

state assessment test scores; 

demographic and census data; 

longitudinal individual student data; 

attitude surveys of students, teachers, parents; 

records of attendance, transiency, vandalism, etc.; 

4. SPECIFIED INFORMATION DELIVERY PROCEDURES: 

formats - printouts, written reports, oral reports, graphic 

presentations, individual and small group briefings; 
cycles - periodic, coordinated with other activities,* as needed; 

5. SPECIFIED MONITORING OF SYSTEM FUNCTIONING AND OF SYSTEM USE: 

informal feedback; 

ad hoc or standing committees reviewing information inputs, 

5utput$;V 
records of system use; 

supervision of subordinates by superiors, peer review; 

6. TRAINING FOR USERS IN DATA-BASED DECISION MAKING AND IMPLEMENTATION: 

in asking questions of the data; - - 

in interpreting test scores; 

in alternative methods of raising student achievement; 

in interpreting survey data; 

in understanding implications of trends; 



in Inferring action alternatives from data; 
in deciding among competing alternative^; 
in implementing change; 

7. AVAILABILITY OF RESOURCES TO SUPPORT ACTION PLANNING: 

training' for Individuals such as media or learning specialists; 
budget for release time, substitutes, conference attendance. 

8. AVAILABILITY OF RESOURCES TO" SUPPORT IMPLEMENTATION: 

trained individuals such as media or learning specialists; 
budget for release time, substitutes, conference attendance. 

Description of Existing Instructional Information Systems 

The eight districts in which we did field studies had unique instruc-. 
tional information system configurations. For three of these districts we 
will utilize our eight components to provide a brief snapshot which will 
illustrate alternative forms of instructional information systems. 
District A: Student Achievement Model 

The purpose of this instructional information system is to individua- 
lize instruction* The direct users of the system are teachers and princi- 
pals. Teachers use the "test score reports, the output of-the-system, to 
plan instruction, to place students in classes, to- group and regroup stu- 
dents, to assign remedial or supplementary material s, to communicate with 
parents. Prinicpals use the reports to monitor individual and group pro- 
gress of students, to review teacher activities, to communicate with pa- 
rents, and to share with one another estimates of school progress so that> 
district policy making can be informed by principal input. 

The type of data which the system collects and analyzes are students' 
criterion-referenced test responses. These criterion-referenced tests are 
keyed to a grade-by-grade district curriculum in math, reading, language 
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arts. The tests are administered by teachers on a quarterly basis. Scores 
are reported by objective, by student, by reading group, by. class, by grade 
level, and by school. Turn-around time from test administration to teacher 
receipt of printout is approximately a week. The format of the Instruc- 
tional information system's output is a computer printout and it is deli- 
vered to teachers. by mail. 

In this district, there are many ways to monitor system functioning 
and system use. Learning specialists in each school make sure that the 
tests are distributed, administered, ,and correctly processed. These learn- 
ing specialists al so assist teachers 1n analyzing and Interpreting the 
scores and in making instructional plans based on these Interpretations. 
The principals review all test scores, hold conferences with teachers dur- 
ing the year to discuss individual children, use the previous year's scores 
in making plans for the subsequent school year. Both teachers and princi- 
pals use the criterion-referenced tests and the objectives to which they 
are Indexed in conferences with parents and in between-conference reporting 
of student progress. 

As for the contributing components of the instructional information 
sytem: The learning specialist 1n each school trains teachers 1n the in- 
terpretation of the test scores and in specific action planning and imple- 
*i. ■ ' ■ ■ . - . 

mentation activities. The criterion-referenced testing and curriculum 
coordination is supported by an elaborate multi -level professional develop- 
ment program (PDP) . In this program, teachers are required to attend 
courses where a diagnostic/prescriptive Instructional methodology compa- 
tible with the criterion-referenced testing orientation is presented. 
Between sessions, the PDP coordinator observes in classrooms to make sure 



that teachers 1 applications of the teaching methodology are appropriate. 
More advanced PDP programs are offered based on an annual survey where 
teachers indicate their preferences for coursework. The PDP program, 
including the release time for teachers, the training of substitutes and 
aides/ and additional conference attendance is part of the regular district 
budget.. 

District B: School Improvement Model 

The purpose of this instructional information system is to facilitate 
school site planning decisions about the allocation of resources to meet . 
needs perceived by parents, teachers,- and students. The primary users of 
the system are school site councils, parents, and teachers, who divide 
themselves into subject matter committees to make plans for subsequent t 
school ^years and to monitor the implementation of previously-made plans. 
Principals are secondary users as are teachers not on the school site 
council . 

The uses to which the data are put include the identification of sub- 
ject areas in which students need additional attention, determination of 
the allocation of ^discretionary school resources for identified school -wide 
needs, analysis of the opinion and attitude data from parents, teachers, 
and students in conjunction with student outcome data from standardized 
norm-referenced tests. 

This district, on a once-a-year basis, administers a standardized test 
of basic skills. The printout is received back from the test publishers by 
school, by subscores. Further analysis is done by the district office and 
is made available to the school site planning teafl. In addition, the dis- 
trict has developed & parent and a teacher attitude survey, sent out once a 
year, collated by the district, organized in graphic format, and 



distributed to each school site council. Each school, furthermore, 
develops and distributes a "Smily Survey" to assess student attitudes 
toward particular subject areas- 

The central office of the district provides to the school site 
council written reports with data not only from the current year but from 
previous years. When -the system was in its infancy small group training 
sessions were held; district officials say that they have subsequently . 
become unnecessary as new school site members are socialized into the pro- 
cess by more experienced colleagues. The distribution of the reports fol- 
lows an annual cycle. The tests are administered in February, the surveys 
go out in March, the information is collated and fed back to the school 
site councils in April, decisions are made in May, plans are implemented 
starting in September, the school site council updates the timelines for 
the plans as the school year proceeds, school site council monitoring of 
the implementation of aspects of the plan occurs at meetings throughout the 
winter. The cycle then repeats itself. 

As for contributing components: Training for teachers and parents 
when the system was first installed included group process skills, conrnuni- 
cation skills, decision making skills, skills in interpreting test score 
terminology. Such training is no longer provided by the district routinely 
although it is available on an as-requested basis. The district releases 
teachers to engage 1n school site planning. The resources for implementing 
the action plans made by the site council come from the California School 
Improvement Budget and have been regularly available over the past five 
years. % 




- 46 - 



Di strict C; Staff Development Model 

The purpose of this instructional information system is to enable cen- 
tral office staff to tra'" teachers and principals in those subject matter 
areas in which students demonstrate deficiencies. The primary users of 
this system are the staff development, curriculum, and supervisory person- 



nel in the central office. Indirect users are principals and teachers. 
The uses to which the information is put are primarily planning and con- 
ducting ongoing and summer staff development activities which either train 
teachers in how to instruct students in a particular area or encourage 
teachers to develop new text or supplementary materials. The Information 
fed Into this* system comes primarily, from a state-wide assessment test -~™ 
which compares school -level student achievement across the state. The 
press and the school district receive from the state the printouts of the 
scores organized iri high-low order of school attainments— Subsequently , 
district officials receive more precise score breakouts. These data are 
supplemented by newly developed district-wide utilization school profi- 
ciency tests. In this district, there is no explicit monitoring of system 
functioning and- system use. 

As to contributing components: Since the primary users are central 
office personnel, there is limited need for training for them in decision 

making and implementation. Substantial amounts of district resources, both 

i ... . ■ ■ / 

1n terjms of time and money, are made available to support action planning 

and the implementation of staff development activities. / 
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W hat to Consider When Developing a District Instructional Information 
System 

As we noted previously, instructional information systems are in a 
sense a selected educational application of management information sys- 
tems. As such, we should look at what has been learned from those who have 
used and-studtsd^he-useful^ systems in other 

organizational contexts. 

Because of space limitations, we will only summarize some of the major 
shortcomings users of management information systems have identified. 
Users complain that they: 

° do not understand output 

- .^ A:^ L _ 

0 do not get accurate Information 

° do not get information that provides them with the type of analysis 
they need 

Systems designers and managers complain that - 
0 costs and development time are high 

° there 1s difficulty 1n keeping the system feasible and adaptive 

Organizational analysts observe that MIS systems 

° ..... do not take Into account the realities of organizational 1.1 fe 

° alter the power relationships among. departments, groups, Individuals 

° change the content of various jobs and taiks 

From those suggestions and ourr field work observations, we would urge 
that Instructional Information systems developers strive to: 

1. Make the system attractive, easy to use, Integrated Into the dally 
life of district personnel, principals, and teachers. 

2. Make the system responsive to the users 1 unique and normal styles of 
Inquiry. . * 

3. Make the system helpful to the user 1n formulating problems. as well 
as resolving' them; 1n generating alternatives as well as selecting 
them. 

" • ""• " " : "' ■* r -i i. " ' ■ V" ' " . : • 
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C ~r»rl. . -i district cructional information systems are in an 
ar ^ eve ier ;al sta^ Br IIS may be an idea whose time has come. 
Existing testing, evaluation^ and research activities that are often not 

-related to one another can be int eg rated into a single comprehensive sys- 

tern. What is more, the development of increasingly affordable personal -'. 
computers provides the technology for easily providing instructional ly 
relevant information to wide and diverse audiences. He feel that the 
potential benefits of this emerging information revolution can be best 
realized if school districts begin linking together and ultimately inte- 
grating their data into a comprehensive school district instructional 
information system. The districts in which we have conducted our research 
bear testimony that such systems can be developed and that they can provide 
a very useful tool in building'and maintaining an effective instructional 
program. 
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THE IMPORTANT DISTRICT ROLE IN EDUCATIONAL REFORM 

Richard C. Williams and Adrianne Bank- 

Background . 

The report of the National Commission on Excellence in Education pro- 
vides the most recent confirmation of a growing national uneasiness about 
the state of American public education. Describing the present public edu- 
cational system as drowning in "a rising tide of mediocrity," the report 
implies that no one has been "attending the shop" and that we (eduators, 
^pareats-r-taxpaye^ 

ensure that our educational system will once again prosper. The Commission 
even gives us guidance in selecting the appropriate strategies to follow, 
including: upgrading text books, lengthening the school day and year, more 
homework, and higher teacher pay. 

To those of us who have been working in education or observing the 
national education scene since its reaction to Sputnik in 1957, this has a 
familiar ring. As we recall, Sputnik supposedly shook the American educa- 
tional establishment out of its "progressive educational" dream world and 
brought it back to the realities of upgraded textbooks, lengthened school 
days, more homework, and higher teacher pay. And if we continue to follow 
educational history, through the 1960's and 1970's we find a continued em- 
phasis on basic* skills through such programs as Head Start and Follow- 

t v 

Through. 

For the past 20 years America has indeed focused its attention on 
schools and has poured considerable .amounts of material resources into its 

■ .' "'-m ' : <54 

eric 



- 51 - 

t 

schooling system* even though the commitment may have lagged a bit in 
recent years due to runaway inflation and a sluggish economy. 

Our basic thesis. can be stated thus: the American people have not 
been ignoring their schools nor have the\or those who work in the schools 
ever abandoned their concern for a quality system. If the schools have not 
improved substantially in spite of continuous reform attempts, it is be- 
cause many of the strategies designed to improve the schools have been 
unrealistic — they have not adequately accounted for the complexities and 
dynamics of the schools as they actually function. 

As we review educational reform efforts over the last few decades we 
can divide them into two categories. In one set are what we will call tar- 
geted reforms (focusing on one component of schooling); in the other set 
are what we will call school site reforms (focusing on schools as cultures, 
with linked sub-components). We argue that the targeted reforms, no matter 
what their specific content, have had very limited success in accomplishing 
the intended goal of improving the quality of education. The school-site 
reforms,, we believe, have been more successful. However, their potential 
impact has been blunted because an essential supporting element the 
school district — has not been adequately mobilized. 

The point of view expressed in this article is that the school dis- 
trict, which we define to include the central office staff and the school 
board, has been aVneglected actor in the school reform movement. We 

*Da"vid Tyack states that from '1958 to 1975, the "Federal government's 
role, in education grew tp .Include 66 categorical programs wnile California 
alone initiated 58 refprm initiatives" (Tyack, Krist, & Hansot, 1980, p. 
259). 
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believe that the district structure has the potential for being an em- 
powering agency for constructive system-wide change but 1n recent years has 
not been so recognized or encouraged. To support our thesis, we will first 
look at some targeted reforms that have had limited Impact, then explore 
the relative strengths and weaknesses of school site solutions, and 
finally, from our research, describe how well -managed districts can 
encourage and support good schools. 

Targeted reforms Include legislated and funded changes that attempt to 
Improve schools by focusing on only one component of schooling, for ex- 
ample, teachers, curricula, school management, student outcomes. Examples 
^ of such targeted reforms Include: changing teacher-and administrator 
credential requirements, mandating more homework, requiring standardized 
testing, Installing accountability schemes. The flaw with these well- 
Intended target solutions, no matter how appealing they are to common sense 
or how well-supported by research, 1s that they do not begin to address the 
multi-faceted systemic characteristics of public schools." Over the years, 
many of these changes introduced Into our vast, complex system have dis- 
appeared without a trace or they have been effectively sabotaged. 
Procedures that appear to be dynamic in conception have become symbolic' 
bureaucratic exercises 1n operation. Perhaps an example of California's 
attempt to improve Its teachers will help to Illustrate our point. 

In the early UWO's, with great fanfare, Call for,. introduced a 
teacher accountability scheme embodied in the Stull Act. The quality of 
' classroom teaching was going to Improve because each school district would 
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be required to establish a procedure whereby principal s *or other teacher 
supervisors would meet periodically to evaluate each teacher. Unique to 
thfe Stull Act was the idea that teachers' evaluations should be based on 
student learning. No more would ^teacher evaluations be based on irrelevan- 
ctfts such as the neatness of their bulletin boards or the pleasantness of 
their personalities; or on the subjectivity of principals 1 judgments. \ 
Teachers whose pupils did not make expected gains were, by definition, | 
teachers in need of assistance. Presumably, 1f the teacher's pupils re- 
mained resistant to learning, he or she. could be dismissed for 
incompetency. 

Now, after more than a decade, it 1s apparent that the~Stull Act has 
not had its anticipated beneficial impact. In district after district the 
Act has. been diverted into a routine set of pro forma paperwork exercises. 
Teachers talk about being "Stulle'd," a process to be endured 1f it cannot 
be avoided. With hindsight, several flaws In the Act's provisions are 
self-evident. Its modified Management By Objectives (MBO) approach carried 
only sanctions — no rewards. If a teacher's students performed satis- 
factorily, everything stayed the same. If a teacher's students did not, he 
or she was ultimately subject to dismissal. The Act Implied that student 
learning outcomes were primarily the result of adequate or Inadequate 
classroom Instruction. Teachers knew this to be an unfair assumption about 
the power of teaching and teachers. They knew that myrladMnfluences out- 
side the classroom, e.g., home Hfe, peers, nutrition, language, transi- 
ency, affect each student's learning and tire collective learning of the 
entire classroom. Finally, the legislation was based on an erroneous 



belief that there were valid, reliable measures that could accurately 
assess student learning. These major conceptual flaws were compounded by 
problems in execution. When the Stull ^Act became law, it became apparent 
that few principals or supervisors could provide instructional supervision 
and assistance to teachers 1n the different subjects taught. Is it any 
wonder that principals and teachers immediately began to subvert such an an 
unworkable reform, -even one with honorable Intentions?. 

While complying with the letter of the law, they violated the spirit. 
Teachers set Instructional objectives knowing that their "students would 
very likely achieve them. Principal's tadtly^went along with the charade. 
And, to no one's surprise, students for the most part achieved the objec- 
tives as they had been written. The prevailing perception among school 
people, then, is that the Stull Act has had little positive impact' 1n 
Improving California's schools. But everyone knows that 1t has generated 
an annual flurry of paperwork absorbant of time and energy which might be 
better spent elsewhere. . 

We could provide other examples of targeted reforms. Most of the 
educational fads of the 60' s and '70' s were of thls^type: e.g., the push 
for better currlcular materials, the swing to open learning centers, the 
expectation that differentiated staffing might work. Our point 1s that 
targeted reforms haven't worked 1n the past and they are unlikely to work 
in the future because they do not take Into account the realities of public 
schooling as complex, dynamic, decoupled systems. 

School site solutions . Educators and legislators, stung by these 
failures, have begun to formulate more sophisticated approaches to 
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educational reform based on the view that the school site-is a culture 
whose subcomponents are linked to one another. Reform then becomes an 
ongoing annual process whereby the principal, teaching staff and community 
work together towards assessing the school's needs, determining appropriate 
solutions, and Implementing and evaluating the results. 

Among the early research into this site-specific problem-solving pro- 
cess was the I/D/E/A study in educational change sponsored by the Kettering 
Foundation. Between 1967 and 1972, 16 elementary schools formed a self* 
help "League of Cooperating Schools" (Bentzen, 1974). Each school, bol- 
stered by a core group of League-affiliated university consultants and by 
support from one another, -undertook^^^^ 

year period of time, some of those schools showed remarkable courage — 
transforming themselves from dull routine places into dynamic, exciting 
. learning environments alive with new ideas and programs. 

Subsequently, several state and national educational programs have 
utilized a similar school sfte approach, e.g., ESEA Title I, PL 94-142, and 
California's School Improvement Program (SIP), California's 1977 SIP 
program required that schools organize school advisory councils to plan and 
Implement educational programs based on documented assessments of need. 
SIP provided funding for start-up and for implementation.' Recent studies 
have shown that* even allowing for the wide variations in how -schools 
carried out the planning process and used the needs surveys, school 
improvement has Indeed occurred (Berman, 1982). 

School site action planning has been an attractive model for educa- 
' tional reform. It takes into account the unique characteristics of 
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Individual schools and gives major actors such as the principal, teachers, 
parents and, in secondary schools, students ownership of the process. It 
is a mechanism whereby schools address their own needs in their own ways. 
Thus, a school where children have reading problems can marshal its re- 
sources and energies to address that need; the school, across town concerned 
about drug use can direct its attention and funds to that problem. 

We applaud the various school site reform strategies. They are vastly 
more realistic than targeted reforms • But we believe that the school site 
model also has limitations: Our main concern 1s that 1t Ignores the pres- 
sures on the school coming from the larger social and political environ- 
ment. It assumes that-sehool^^eed-no^u^^ 

support structure outside themselves. It assumes that they are in charge 
of the major aspects of their own governance. 

But individual schools have limited control over the size of their 
enrollments and of their budgets. The former is influenced by population / 
trends, the latter by legislative actions. School managers have only 
limited power to hire and fire their own personnel: they are constrained 
by pre-existing legislation, administrative rules, or union contracts. 
Publicly supported schools are subjected to nation-wide or state-wide 
legislative, judicial, social, and political forces many of whose mandates 
are misaligned with the instructional mission of an individual 
institution. 

The *f ore, we believe that individual schools are not arge enough or 
strong enough to initiate and sustain improved instructional functioning by 
themselves, even when they make heroic efforts in this direction. They 



need continuing financial and technical and psychological support from some 
larger entity 1n order to sustain their own renewal efforts. 

I/D/E/A's League of Cooperating Schools 1s a good example of what 
happens when an ad hoc support system assembled for , the specific purpose of 
providing such financial, technical, and psychological support disappears. 
For five years, League consultants and staff played a critical role in 
encouraging each school to identify problem areas needing attention. The 
League provided services on request, e.g.: training for principals and 
teachers, coaching in group dynamics and problem solving, Intellectual 
stimulation and provision of ideas. With these enabling and empowering 
services, many schools -developed and improved remarkably. However, when 
I/D/E/A funding ended, the League disbanded. And schools slowly began to 
transmute back Into what they had been before the project began. When 
principals and teachers who had been leaders in the reform effort left, 
they were replaced by others without their training or* commitment to - 
change. There was no external support system which could sustain the 
Innovations and counterbalance the personnel changes. If one were to visit 
those eighteen schools today, one would find most to be rathier ordinary 
elementary schools,, mere shells of their earlier innovative selves . 

From our experience with the I/D/E/A project and from research on 
school change (Herriott A Gross, 1979; Lehming 4 Kane, 1981; Rosen&lum 4 
Lowis, 1981), we derive an important Insight. Missing in both the targeted 
reform and the .school site approach is ongoing support coming from a 
stable, sanctioned, organizational entity of which the Individual school is 
but a part. In the I/D/E/A prbject the larger structure was, 
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temporarily, the League itself* In the everyday world of schools that 
permanent entity should be the school district. 

A common reaction by those teachers, parents, and community groups 
interested in school improvement is that the district offite is an obstacle 
to be overcome, a bureaucratic morass where good ideas get buried or 
subverted. When budgets must be reduced, central office administrators are 
often seen as an easy target. We argue, in contrast, that urgent 
consideration be given to adding onto any school site reform strategy a 
role for school districts. Academic, popular, and political attention 
should be turned to a most obvious locale for initiating, coordinating, and 
sustaining education change — the school district. 
The School District Role in Educational Reform 

The recent history of programmatic change in education gives insight 
into the current, school district role in educational reform. At the turn 
of the twentieth century, a key political goal of educational administra- 
tors was to centralize control of urban schools, to standardize publl edu- 
cation, and to vest most decision making in appointed expert superinten- 
dents. As these progressive administrators redefined the concept of demo- 
cracy, the school systems they constructed were literally hierarchical and 
shielded from lay influence (Tyack, et al., 1980). Many of the ideals and 
achievements of the progressive administrators came under sharp attack 
during the I960 1 s and 70's.. The reform generation, starting with Sputnik 
in 1958 and lasting until the mid 1970' s, increased the federal govern- 
ment's rol<* in education to include 66 categorical programs 7ft -J s created 
a climate of heightened factionalism over which schooling Junctions are 
most important. • 
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Despite the reform rhetoric, regulations by the state, strictures of 
accrediting bodies, the influence of testing agencies, and bureaucratic 
inertia often' inhibited change.- One result of the new politics of educa- 
tional reform, however, was an increase in regulation of local districts 
and new pressures for lay participation at the local level (Tyack, et al., 
1980). Federal and state laws mandated school site councils. Large school 
districts experimented with decentralization and community control. 
Teacher unions grew in number and influence and adversarial relationships 
became common in school systems. 

In the 1980's, the educational reform picture is changing again. 
Issues such as declining enrollment and tax revolts have surfaced. Current 
strategies used by school districts to improve instructional programs re- 
quire the ability to coordinate complex subsystems within the educational 
organization. 

It is clear that districts differ substantially from one another not 
only in terms of* size and level of resources, but also in their administra- 
tive philosophy in regard to instruction. Some hands-off districts regard 
classroom teaching as the responsibility of the teacher subject only to 
whatever supervision the principal wishes or is able to provide. Other 
districts centralize scope and sequences of major subject areas, limit 
schools to one or several options for texts, test all students on their 
achievement on milestone objectives, and have district-wide mechanisms for 
creating strategies to remediate student deficiencies. ' Such a ; highly 
coordinated approach .to instruction is rare, takes a long time to develop, 
end & . ikely tv be appropriate only in districts where particular 



conditions are extant (Williams & Bank, 1982). These include the presence 
of "idea champions" who work with a stable core of staff 1n a community 
environment wi ich 1s not embroiled 1n turmoil or rapid change. 

We are not suggesting such a centralized system as the way for all 
districts to perform. Neither are we advocating the hands-off approach. 
Rather, we are advocating that the district office attend on a systematic 
and regularized basis to the development of situation-specific coordinated 
ways to Improve student learning. It 1s our view that attempts by the 
district to do their own targeted reform — such as adding teacher training 
programs, testing programs, buying new books — will fall, just as similar 
reforms fall when mandated by state or federal agencies, because they 
address only a corner of the educational tapestry. Similarly, we believe 
that districts which only pay Hp service to school site reform, without 
providing back-up encouragement, technical assistance, and training from 
the central office will also fail. 

This implies that there are two crucial criteria for those 1n the cen- 
tral office who want to move their districts towards Instructional excel- 
lence: they must consider the connection between all the parts of their 
complex educational organization; and they must think of their efforts as 
part of a long-term sustained effort ~ to be modified 1n the light of 
changing conditions but not abandoned as soon as the public turns Its gaze 
elsewhere. 

He have come upon several districts which have such long-term 
strategies for supporting on-going , Incremental Improvement 1n their 
schools. These districts did not start with a blueprint or prescription 



for instructional improvement. Instead, although operationally quite 
different from one another, each started when one or several individuals 
— who cared, and who had clout — built a district constituency with a 
commitment to instructional excellence and had the confidence that they 
could move their system towards that vision. 

Each of the four districts to which we are referring began by trying 
to understand what their current situation was at the moment. In both 
formal and informal ways — that is, by looking at test data, program 
descri ptions , instructional activities , teacher and parent surveys , needs 
assessments, corridor conversations — those in the central office who h^d 
made a commitement to educational- excellence first gathered information to 
identify strengths, resources, and energies as well as trouble areas in 
need of fixing. 

In District A, this situation assessment led to the development over 
an eight-year period of time of a highly Integrated criterion-referenced 
testing system linked to a district-wide scope and sequence in math, 
language arts, and reading. A district-funded professional development 
program provided teachers with methods to remedy student learning 
deficiencies. A learning specialist in each school helped with classroom 
management details, the principal in each school led the annual planning 
and feedback sessions and monitored classroom progress. 

District B, with the same commitment to instructional excellence, used 
a different strategy. They vested decision making over instructional 
operations in the hands of a parent/teacher school site council . On an 
annual cycle, this council received information culled from questionnaires 



and test scores by the research and development office. The Information 

was formatted so the council had a snapshot of their situation which they 

could compare with 'previous snapshots of their school. Problem-solving 

meetings in each school in April generated an annual plan to start iti 

September which had targe* dates throughout the year for specific actions. 

District monitoring of the target dates kept upthe pressure on the schools 

to improve according to their own plan. 

Di strict C, a 1 arge di strict spread oyer^many. miles. of urban and rural 

communities, used their situation analysis as the catalyst for generating 

district-wide consensus on both student outcome and teaching method goals 

for their entire system. Teachers, principals and central offlce^admini- 

stratcrs wrote their own job descriptions to reflect these goal positions 

and became willing to hold themselves and others accountable for perfOr- 
fe 

mance. , * 
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District D, a medium size heterogeneous district with majjy minority 
groups, brought principals together to analyze their own schools' scores on 
a mandated state assessment test. This Ted to the proposing of Individual- 
ized school remedies, some of which called for Increased Instructional time 
to be spent on a particular subject area, supplementary materials to be 
purchased, and additional staff training. The district stimulated the pro- 
cess, made available the funds, and organized the staff development. Not a 
one-shot deal , this process of taking stock goes on every year, supported 
by the district office (Bank & Williams, 1981; Williams & Bank, 1982). 

We know of other districts who have created their own versions of 
coordinated Instructional Information/action systems to support their 
visions of Instructional improvement (Bank & Williams, 1981). Such 



systems do appear -to be within the capabilities of most district ,off ices, 
especially as computer use is becoming more accessible. 

We believe that, the time is ripe to" provide to those districts who 
have the will but don't quite see their way clearly, with support, encour- 
agement^ and technical assistance from the larger educational environments 
of which they are a part. For example^as state departments of education 
put together reform packages^ in response to the renewed public call to 
excellence, as universities ponder the partnerships they might, develop with 
school districts (Goodlad et al., 1983), as county offices provide their 

technical services, we believe there should be cognizance of the district 

M - 
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officers a neglected resource. Research has suggested factors which 
characterize effective schools (Edmonds, 1979; Brookover, et al., 1979; 
Rutter, et al., 1979; Clark, et al., 1980, Murnane, 1980). We believe that 
effective schools, 1f they are not to be accidental and evanescent must be 
supported and ma1nta1ned\by effective districts. It 1s essential that' we 
learn more about the characteristics of effective districts and how to 
bring them Into being. We call for heightened political, academic, and 
public awareness of the district as an Important agent 1n educational 
excellence so that the process of creating' effective districts can move 
forward. 
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- THE DISTRICT ROLE IN INTRODUCING MICRO-COMPUTERS: 
' A CONTINGENCY APPROACH 

- by 

Richard C. Williams, Adrianne Bank and Carol Thomas 

Introduc tion 

There are, among those in education, computer optimists and computer 
pessimists. Computer optimists can visualize schools of the future as part 
of large scale networks allowing students and teachers access to informa- 
tion of a quality and quantity never before possible. They see computers 
rectifying the resource disadvantages of small schools, meeting the needs 
of minority populations, encouraging problem solving, creativity, and Indi- 
vidualized instruction. Computer pessimists, on the other hand, see read- 
ing and writing devalued as more time is spent with computers and less time 
with books, greater personal isolation as learning occurs primarily through 
interaction with machines rather than with other people, a widening gap 
between the rich who have computer access and the poor who do not. 
(Coburn, et al., 1982) 

But whether one 1s an optimist or a pessimist about the future Impli- 
cations of computers, as a school or district administrator, one must be a 
computer realist . According to Market Data Retrieval , October, 1982 fig- 
ures, based on their annual telephone survey of all U.S. districts, over 
24,000 public school s now use microcomputers in instruction, up 60 percent 
from the previous year with the fastest growth rate occurring in elementary 
school s. - 

Of the 15,314 districts" in. the U.S., 9,245, or 60.4 percent, had 
microcomputers in 1982 as compared with 6,441 a year earlier. The rata of 
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growth was highest in the smallest districts. By October 1982, 52 percent 
of these small — under-1, 200 students — districts had microcomputers 
while almost three-quarters of the districts with over-10,000 students had 
them (Market Data Retrieval, October, 1982). 

The rate of growth in schools and school districts' acquisition of, 
microcomputers is phenomenal and is expected to continue. But the current 
statistics on the availability of hardware may be misleading. The National 
Center for Educational Statistics reports that computers were used by an 
estimated 4.7 million students during the 1981-82 school year, averaging 
ing over 9 hours a year of computer access for each student. Differences 
in amount and type of use were 1 by grade. High schools cite computer 
science as their major use in instruction; junior highs use terminals for" 
remedial instruction, enrichment and computer literacy. In elementary 
schools, terminals are used mainly for enrichment, remediation and basic 
skills instruction (National Center for Educational Statistics, 1982, 
p. 2). 

9 

There is great variation, then, in the availability of personal com- 
puters in schools and in the uses to which they are put. There is also 
variability in the role which district offices play in introducing compu- 
ters Into the educational setting. 

Some districts, especially large districts or those with strong cen- 
tral administrations have adopted a highly centralized approach to Intro- 
ducing computers. Here, the district directs the process of selecting, 
funding, and placing microcomputers in schools, usually with some input 
from principals or teachers. The advantages of such a centralized approach 
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include 1) the development. of conveniently located and deployable expertise 
for training and troubleshooting in many schools; 2) the increased capa- 
city to coordinate hardware, software, and training; and the added clout 
the district has when negotiating with vendors on 'price, service contracts, 
and software when they purchase in bulk. Disadvantages of the centralized 
approach include diminished teacher "ownership" of and enthusiasm for both 
the hardware and software, less flexibility in accommodating specific . 
classroom needs for particular kinds of twdware or software, lack of know- 
ledge at the central office level. 

Other districts have, either inadvertently or intentionally, adopted a 
grassroots approach to introducing computers. In these cases, computer 
buffs among the teachers learn as much as they can, find their own or apply, 
to the district for funding, and use their, own computers in their own 
classrooms in their own ways. Their enthusiasm, 1t is assumed, will spread 
to other teachers, who will then become a critical mass who will eventually 
come together to form a school -wide plan. 

Advantages of this approach Include Its low cost to the district for 
educating its own personnel and grappling with individual schools 1 prob- 
lems, and the. natural ^spread of the Innovation because of enthusiasm and 
Individual initiative. Disadvantages, however, may be serious: much money 
may be spent on hardware and software while only a few children will learn 
particular skills, and these skills either may not be picked up 1n subse- 
quent grade levels or subject areas or may be unnecessarily repeated. 

Between the extremes of a highly centralized and a grassroots, 
approach are many Intermediate approaches. Each district where the 
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computer issue has arisen t ~ and we should note that close to 40 percent of 
the districts have not yet grappled with the situation seems to muddle 
through, formulating its own responses In reaction to various kinds of t 
pressures. ^ 
The Growing Importance of the District Role 

Our argument here is that the district central office, along with its 
school board, must take ser1ously\its role in introducing microcomputers 
Into its schools. Mistakes are becoming Increasingly costly. Some dis- 
trlcts have rushed out to buy microcomputer systems and found, unhappily, 
that the system they bought will not e»:rttinue to meat their needs and that 
their instructional programs are not wall served by the system they have 
purchased (Thomas & McClain, 1981). • 

The unfairness to students of leaving issues of computer access to 
chance is becoming more apparent. A survey conducted by Market Data 
Retrieval (1982) found that , school microcomputer use is associated with 
wealth of the district —,"80 percent of the nation's 2,000 largest, richest 
high schools used microcomputers, while only 40 percent of smaller, poorer 
high schools had them (Upkin, 1983). Access to micro-computers is also 

dominanted by male students. A survey of 10 Mew Jersey high schools offer- 

/ 

ing computer courses revealed a consistent dominance of male enrollment, 
slightly more than 60 percent. Studies of California schools report a 

similar trend/(Bakon, et al., 1983). In addition to computer access, the 
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issue of equity is also reflected 1n how schools use computers. When com- 
puters are used 1n suburban schools, it is often 1n the context of program- 
ming and computer awareness, kiu., ^nputers are used in less affluent 



1nner-c1ty or rural schools, the use is more likely to be for drill and 
practice and remediation (Field & Kurtz, 1982; Lipkin, 1983). The desir- 
ability of having a coherent computer literacy scope and sequence, analo- 
to that in reading, math and language arts, is dally becoming more 

evident. 

There are, in short, many issues that are too large and too complex 
for individual schools to resolve each in their own manner. In the current 
world of educational computer use, effective, districts are essential for 
effective schools. A brief summary of some of the issues with which a dis- 
trict must eventually cope is Included in Table 1. . We have grouped. these 
issues into categories: hardware acquisition/fiscal Issues, software 
Issues, management issues, staff development Issues, and Instructional 
issues. The issues in each category have been organized according to major 
policy questions and operational planning questions. 
* Contingency Approa^.i to u. strict Involvement with Computers 

As noted above, many districts have responded reactively to the 
rapidly expanding availability of relatively Inexpensive computers and 
programs that can be used for managerial and instructional purposes. 
Whether centralized or grassroots 1n character, their approach might well 
be labeled a "non-planning strategy." 

There are a number of understandable reasons for the prevalence of 
this approach. School districts, like many Individuals and other organiza- 
tions 1n the public and private sector, are unsure about how to asses? 
potential value of an Eroding" technology. And there are other reasons 
related to the marketing of computers. For example, many computer vendors 
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Table 1 

Issues in Need of District Attention 



Issues 



Policy/Framework Questions 



Hardware 
Acquisition/ 
Fiscal 
Issues 



What criteria/guidelines should be 
established for hardware acquisition? 
What percent of the budget 
should be allocated for software purchase 

anc!j1ntenance? 

Should a sin^e computer system be used " 
for both instructional and administrative 
purposes? 

What percent of the computer budget should 
be allocated for software purchase and 
maintenance? . 

What resources' are available for personnel 
costs associated vri th hardware use? ° 
What Inservlce training budget allocations 
should be made? ' 

What' strategies should be used by educators 
1n dealing with computer vendors? 



Operational Planning Questions 



'What successes/failures have been experienced 
by other districts with specific hardware? 
What is equipment's reliability? 
What maintenance warranties and assistance 

will vendors provlde in installing and 

servicing the equipment? 
What peripherals are available for specific 
hardware and provided by the vendor? ' 
, What expansion options exist? 
What training will the vendor provide In the 
operation and programing of the hardware? 
What size machines and/or memory are required 
to run the programs needed and achieve 
computer use objectives?' 
„ What software is available and at what'cost 
In relation to the characteristics of 
hardware? 

What are the estimated costs for hardware, * 
software, maintenance, facility preparation, 
and staffing needs for each application? *. 
What strategies should be used for financing 
computer acquisition? 
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Issues 



//Framework (juestions 



Under what conditions should districts 
undertake software development? 
Should the district operate a software 
library? 

What Is district policy relating to copy- 
right issues for' purchased and teacher- 
developed programs? 

How and by whom should software be located 
evaluated; and acquired? , ; 



Operational Planning Questions 



0 How: w results of software evaluation be 

disianinated? 
0 Do the software cassettes or discs include 

documentation? •• 
0 Is the software program educationally sound? 
0 ( How can computer software be integrated 

'with other instructional activities? 



What role will other educational service 
agencies and "groups have in the district- 
framework and plan? 

How will the district judge If their compu- 
ter implementation program is successful? 
How should resources be allocated to 
ensure equal educational access and use 
of computers? 

What security precautions should be taken? 
What phasing-in strategy should be Imple- 
mented for the district's computer plan? 



What implementation strategy and timelines 
are' needed for etehtary and secondary levels 
of the district? 

Should schools have central Ized placement 
or Individual classroom/deparUnent 
placement of computers? 
What strategies can districts use to 
encourage female students in computer use? 



what do teachers, principals, and other dis- 
trict staff need to know to use computers? 
What teacher certification requirements 
should be established, if ahy? 
Who should conduct and evaluate the conn 
puter training and what type of follow-up . 
assistance will be provided? 



Will the district develop staff to be local 

computer resource persons? 

What computer training, both preservice ' 

and inser^ce, should be required for 1 

teachers and administrators? 

What strategies should be used to allocate 

time for staff training and hands-on . 

'computer experience? 



Issues 


Policy/Framework Questions 1 


Operational Planning Questions 


InstrtKMi 
Issues 


0 k „ tiN"M computers have in the' 
sciiooi,^ fu't^-assl^id instruc- 
tion, computer literacy, conputer 
progranmlng? 

, 0 Should all students meet minimum computer 
competency requirements? 

0 How will tne instructional role of 
teachers change with increased computer • 
use? 


0 What kind of social problems are being intro- 
duced into school: along with computers? 

0 How can the district ensure equity In computer , 
use, especially higher ) r and creative uses? ■ 

0 What are reasonable ml sidelines for 
student computer use? 

0 Is there a specific need for a "computer 
literacy" curricula? 
. 0 What are appropriate educational goals and 
curriculum materials for computer literacy? 
' 0 How can the teacher overcome the constraints 
of using Individually-oriented computers in 
the context of a group-based Instructional 
organization? 

\ * 
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provide free or low-cost introductory offers to school districts in 
.hardware or software or staff training in order to get districts to make a 
long-term commitment to the vendor's brand. School districts operating on 
meager financial resources find it difficult to refuse the hook hidden in 
this sudden technological largesse and they purchase before they plan. 
Another reason for non-planning is avoidance: the level of uncertainty and 
ambiguity is so high that central office staff don't know where to begin in 
devising a comprehensive strategy. The hardware and software is constantly 
changing; is unfamiliar to many who would potentially benefit from its 
availability; threatens some who think they don't want to or can't learn 
about it. A further psychological complication is created by students who 
seem to know far more about and have far greater aptitude and appetite for 
this new technology than do their teachers. 

. To some administrators, the'logical response to this problem 1s to 
begin with a rational planning model, following a series of sequential 
steps that would include: carefully defining the district's objectives as 
regards computer use; determining those steps that would have to be taken 
by various district components, e.g., teachers, district administrators, 
principals, in order to accomplish each objective; establishing time lines 
and sequences to be followed; determining ways to evaluate whether specific 
objectives, had been achieved; applying corrective actions 1n instances 
where objectives had not been met. 

L1near r planning can be an effective tool to help organizations achieve 
specific goals when there is a common knowledge base, where lines of 
authority are clearly defined, and where there are the resources to 
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carry out the implementation sequence; we doubt, however, that linear 
planning is an appropriate tool for questions of computer selection.. 
School districts lack sufficient knowledge about or control over important 
factors that must be accurately estimated in order for a' linear model is to 
work. For example, school districts are subjected to shifting forces 
outside of their organizational boundaries ov^r Which they have little 
control , e.g. , political support in the community, changing population, 
externally mandated strategies in key administrative and instructional ° 
areas, and uncertain financial resources. Given these conditions, and the 
rapidly expanding computer technology, we think it a waste of time to try"* 
,to determine exact goals and the means to accomplish ttfem. By the^time 
such a comprehensive plan is devised, it is' likely that conditions will 
have changed so as to make the plan obsolete. 

Under such conditions of uncertainty and change, we reject both "no 
planning" and "linear planning." We suggest instead the use of an inter- 
mediate scheme which we will refer to as a contingency planning approach . 
This approach suggests that districts 1 planning be ongoing, incremental, 
adaptive and self -correcting. 

While traditional planning is based on events that have a high proba- 
bility of occurring, contingency planning takes into consideration other 
likely conditions, which, if they actually occured, could create serious 
difficulties for aschool district. A contingency approach prepares one to 
take specific actions when an event or condition not planned for in the 
formal planning process actually does take place. It therefore eliminates 
uncertainty and time delays in making responses, and makes responding to 
the unpredictable a reasonable part of daily life. * 
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A contingency approach identifies issues of concern (e.g., "what if" 
questions) and estimates the probability of their occurrence (Stei ner, 
1979). Both the .degree of criticality and the degree of probability must 
be considered. Alternative strategies to deal with the possible occurrence 
of these events are identified and considered in terms of the anticipated . 
nature of the events arid the district's capabilities and constraints in 
dealing with. them. The. result may be a decision by district staff to take 
some advance "damage control 11 actions as well as to identify potential ' 
strategies to be followed at the time of the events. 

•A contingency approach may describe "trigger points K or those warning 
signs which would signal the imminence of the events for which contingency 
plans have been developed (Steiner, 1979). In some cases, 'the trigger 
point might be the event itself, but in other cases the point at which some 
action should be taken is less clear. 

For example, using a contingency approach, districts should begin or 
continue to become knowledgeable about a wide range of computer-related 
topics from technology to staff needs, attitudes, and purposes. At the ^ 
tame time, the district should become aware of present use^s of computers 
and start to imagine alternative arrangements that could accommodate the 
district's activities to the technology's demands. 

Armed with such data, the district should, at the same time, s identify 
the optimal* dates by which it must make critical decisions regarding, what 
computers to' buy, when they (should be bought, who should use them, and who 
should have them. In other words', many of the district's future plans and 
actions will be contingent upon the unknown opportunities that will' be \ 
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emerging at some unknown point in time. It is a complex task to decide not 
only how, but when, to act. 

Components of a Contingency Approach 

Gearing up for computer use in the district can be viewed as occurring 
simultaneously in three areas: I) doing a situation audit (external and 
internal environments); 2) generating support; 3) formulating district-wide 
policy; and 4) developing an ongoing operational plan to facilitate deci- 
sion making. , 
1. Doing a .Situation Audit 

The tefyi situation audit refers to' a systematic analysis of data, 
past, present^ and future (Steiner, 1979)^ Such an audit provides the base 
for planning computer purchase and use. pte potential range of topics 
covered in a situation audit is wide — anytojiig of importance in the in- 
ternal and external Environments. A major objective of the situation' audit 
is to identify and analyze the key trends, >fofces, and phenomena that have 
a potential, "effect on the* formulation and Implementation of a framework for 
district computer use. The situation audi t_^rt-$o provides a forum for shar- 
ing and debating divergent views about relevant tissues regarding' potential 
changes. We discuss the situation audit ^Y^termp of ah internal inventory 
and an external resources listing. 

. An internal inventory . In order to develop an effective district 
framework, administrators need to know What is already occurring in the 
community, schools, and homes of students enrolled in the district. 
Through surveys and interviews, baseline information can be collected 
regarding what equipment is now available, how much it is now used, what 
resources and skills there are at present in the district. 
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Detailed information is needed on the district's current inventory of 
types of hardware, maintenance problems and their costs, support from ven- 
dors, the extent to which existing hardware is compatible and expandable. 
Districts should know what software has been purchased, where it is stored, 
how much it has been used. In addition, the district will need, to know 
who, at each school, is managing the use of the computers, how they are 
being used, and ror what percentage of time. In California, one district, 
inundated with a variety of microcomputers, "conducted a survey to determine 
what equipment existed in their schools.. They found that during the past 
few. years each secondary school department had been acquiring its own 
.equipment to meet specific needs. This piecemeal acquisition was now 
creating problems since schools had bought different brands (Stremple, • 
1983) . 

Staff in the district also can be surveyed to determine who has skills 
for operating what equipment and software, who can program in various com- ^ 
puter languages, who can be a trainer of. trainers, demonstration teacher, 
or software evaluator. Parents of students enrolled in the district can be 
surveyed to determine if a computer is in the home, what type, and if it-is 

used by the student. 

The district data base should also indicate what information is 
already being systematically collected by the district about existing 
instructional programs, demographic profile, student achievement data, 
financial transactions, etc. 

Finally, an inventory can assess teachers' and students^ attitudes 
towards computers to discover those who are likely opinion leaders and 
those who are not. 
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An external resources listing . In addition to internal resources, 
there- are many groups and agencies external to the school district that 
might provide assistance to district staff contemplating computer use. 
Electronic Learning magazine (1982) conducted a survey, that identified 38 
statewide educator-user groups in 33 states, all of which have the general 
aim of promoting the effective use of computers in the classroom. In those 
states where no statewide groups were identified, most often a special unit 
within the state department of education was filling the role. These 
groups varied in the services they offered, providing a range of the fol- 
lowing activities: cooperative funding, newsletter publication, conference 
organization, resource center, inservice training, software library,^ and 
software evaluation. A few of these user groups have national member- 
ships. For example, school teachers in the Santa Clara County area of 
California formed the Computer-Using Educators (CUE) group which has a 
membership of over 700 people in 19 states (Unseem, 1981). Minnesota 
Educational Computing Consortium (MECC) provides services to Minnesota 
schools and schools in adjoining states. 

Corporations and industry leaders also provide support to school 
districts. Hewlett-Packard in California has fostered industry-education 
ties by having a number of full time_ employees who devote time to improving 
the company's contact with public schools. A committee of top executives 
examines ways the firm and industry can provide more support for public 
education. They have loaned personnel and given equipment to schools 
(Unseem, 1981). A partnership exists between the Washington, D.C. schools 
and Control Data Corporation. Their partnership calls for. the firm to 
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donate $118,000 worth of terminals and spftware and an equivalent amount 
worth of training and administration to the school district. The s^nool 
system will be matching that contribution ( Education Daily , 1.982). 

Organizations also exist that provide services to districts in speci- 
fic areas of computer use such as software evaluation (e.g. ,.M1croSIFT 1n 
Oregon)-; Information exchange (e.g. , .Association for Educational Data - 
Systems); data bases (e.g., Resources in Computer Education [RICE]); and 
newsletters and magazines (e.g., The Computing Teacher , School Mlcroware 
Directory , Software Review ) . 

In Its survey of external resources, the district should become know- 

/ . ■ 

ledgeable about the talents, skills, and attitudes of people living within 

/ " . 

its attendance area such as merchants and Industrial specialists. 

/ • 

2. Generating Support Within District 

This 1s a top-priority. The biggest problem technology enthusiasts 
had a few years/ago was convincing educators, that there was a need for 
computers 1n our schools; today, In many districts, that 1s no longer such 
an obstacle^OHver, 1983). But commitment from groups such as board 
members, /parents, administrators, teachers, Industry and community leaders, 
and otner educational resource agencies Is necessary to build a policy 
consensus. A network of Interested persons can be a continuing support 
/System for services, equipment, or funding to achieve, program goals. 

r ' Successful strategies for generating support for a computer policy 
vary from district to district. Hands-on experience helps. In some 
districts, having computers available for -home experimentation by 
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teachers and principals has been effective. Establishing demonstration . 
sites so that board members, principals, and teachers can have the oppor- 
tunity to see computers in operation and have some hands-on experience has 
worked In other districts (Swalm, 1983). School districts have loaned 
school computers. to parents over weekends and* holidays. A large school 
-district 1n Texas Initiated a. computer project that offered low-Income 

parents and children 12 hours of Instruction, after which parents could , 
. check out computers for home use (Sturdivant, 1983) . Other school dis- 
tricts have organized -computer fairs, computer 'cl ubs; and computer compe- 
titions to Increase public and student Interest (Fisher, 1983). 

Identifying an enthusiastic "Idea champion 11 1n each school can per- 
suade other teachers to consider approaching the computer supporters in the 
district. One Texas district developed a new job role called "teacher 
u technologist" for each school (Sturdlvant, 1983). Resource centers and use 
groups have also been formed to share Information between schools (Useem, 
1981; Stremple, 1983; West, 1983). 

Idee champions 1n districts are also critical to the success. of any 
■< computer use plan. " In some districts, administrators have created formal 
structures to address Issues and allocate resources. For example, the 
Houston Independent School District has a new division called the 
Department of Educational Technology that Is responsible for Implementing a 
district-wide plan for computer use (Oliver, 1983). 
3. Formulating a D1str1ct-w1de Policy Framework . 

A critical process in the Implementation of a district "computer pro- 1 
gram 1s to formulate a framework that will guide the development of an 
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operational plan. A framework allows the district to examine all aspects . 
of computer use and then decide, the best applications for students in the 
district. The development of a framework for a contingency planning 
approach begins with the assumption that each district's planning process 
is unique and allows for a range of expertise, points of view, and 
experience to apply to issues. 

With administrative support, an inter-school committee can be organ- 
ized and charged with the responsibility for developing the district's 
policy framework. If the district wants computers, to be used by all 
teachers, the committee should not be dominated by any one subject area 
(Swalm, 1983). The committee should include representatives from 1nter 0 - 
ested groups while remaining small enough to constitute an effective work- 
ing group. One large district with a committee of 25 members took two days 
to agree on only four goals related to computer Lie, while another district 
committee, with seven members, wrote the entire plan in one day (Fisher, 
1983). 

In deciding upon district policies the committee needs to list the big 
picture issues it will discuss. In doing so, it should decide whether the 
central office or the schools will make the decisions on those issues and 
whether the decisions should be made now or put of f until sometime in the 
the future: 

° Computer use: Instructional applications? Administrative 
... applications? Both? • 

° Criteria for hardware acquisition;. 

0 Software locatlqn, evaluation, and^acqu1s1t1on; 

' 0 Software development; 
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° Maintenance of software library; / 

°, Evaluation of computer use program; - / 

° Development of teacher certification requirements; 

° Development of mininlum computer competency requirements for 
students. ^ 

The task of the committee Is to develop a policy framework of broad 
qoals and an overall timeline. Districts tha/ have successfully integrated 
computers into school programs developed pylcy frameworks that spread 
implementation over several years. West (/1983) found that the best way for 



their district to incorporate computer Literacy Into the curriculum was to 

/ * > 
develop a five-year framework setting goals and objectives In Instructional 

and management applications. Fisher/(1983) suggests that a long-term 

framework is more effective than a one-time plan. According to Fisher , 

having a framework spanning several years signals a continuing conmltment 

by the district and is vlslble^evldence that teachers can become Involved 

in the planning at several stages of the process. A long-term framework 

can 'also aid 1n reducing the fiscal burden In any one year. 

General financial planning should go on concurrently with developing a 

framework. A common error In financial planning 1s to think only about the 

Initial direct cost of the computer facility. Larer and Moursund (1980) 

listed other aspects that should be considered: 

0 The needs assessment and general planning, writing of specifi- 
cations, dealing with vendors, evaluation of bids, supervision of 
Installation, all take time and expertise that require financial 
resources; 

. , ^ site preparation for the facility; s 

0 Computers use supplies and supplies maintenance, both of which mean 
ac*i1t1onal costs; 



° Computers need to be maintained and repaired — a- standard estimate 
1s that for large computers a maintenance contract costs about .75% 
of total equipment cost per month and for microcomputers,, perhaps 
• Z% per month; 

° Large computer systems require operators and a programming staff 

° Teachers need to be trained; curricula may need to be revised; . 
courseware may need to be developed . 

° Software may need to be revised, developed or "acquired. Sqftware 
will also have to be maintained and distributed. 

The goals will facilitate the definition of school-level objectives 
and determine at what grade level and 1n what subject areas each should 
occur. For example, 1n a framework developed by one California school 
district, under the broad goal of programming, modifying computer programs 
was an objective for students 1n grades 6-8 (Fisher, 1983). 
4. Developing an 0n-go1ng Organizational Plan 

Using the policy framework, either the committee or other school or 
sub-ject groups may want to develop more specific operational plans. Acti- 
vities that the committee members might engage 1n to contribute to the on- 
going operational plan could include analyzing curriculum needs, investiga- 
ting and evaluating software, v1 siting programs 1n other school districts, 
attending conferences and vendor demonstrations, and developing staff deve- 
lopment strategies. Such plans can identify specific aspects of each 
school's use of computers. An on-go1ng operational plan may want to state 
Instructional objectives in terms of types of students, grade level , and 
subject areas. -Instructional ' objectives might be some or all of the fol- 
lowing: to develop computer literacy for all "students, to provide the 
elements of programming using BASIC, to provide equal access to computer 
time for all students, to use computer-assisted Instruction for remediation 
in basic skills for Identified groups of students. 
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In Albany Unified School District in California, for example, under 
the broad goal of use/operation of the computer, objectives were given for 
three subsets of grade levels: K-5 students would learn how to operate the 
computer, load programs, and respect copyrights; -6-8 students would focus 
on appropriate computer use, typing, keyboard, and functions; and 9-12 
students would spend time on appropriate programs and vocational use, such 
as word processing, data bases, network, and telecommunications (Fisher, 
1983). r 

In Cajon Valley Unified School District, also 1n California, all of 
the 22 schools 1n the district were asked to submit a statement of assur- 
ances specifying how they would use computers, what their goals and student 
objectives were, how they would evaluate the program, and who would be re- 
sponsible for their school's computer program (West, 1983). 

The ongoing operational plan might also Include objectives and strate- 
gies for staff training necessary to Implement the district computer use 
framework. A school district 1n New York State developed the following 
four inservlce goals: to acquire a functional knowledge of computers for 
educational use, to learn how to integrate computers Into the learning 
environment, to develop the necessary programming skills to facilitate ere 
atlon of software suitable for classroom use, and to acquire the knowledge 
necessary to teach principles of computer awareness (Center for Learning 
Technologies, 1982). -. 

Nalman (1982) proposed the following staff development strategies: 

° Have Individual teachers, already knowledgeable, train others; 

° The school or system can provide Inservlce courses during or 
outside of class times or on inservlce days; 

V k* 

° Push on the state 'department of education and regional centers to 
.r offer computer training; 
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° Some professional associations offer computer workshops at their 
meetings; 

• System can provide release time on a regular basis for teachers to 
take courses; 

• provide sabbaticals for someone in the district to learn and then 
share ^expertise with others; 

° Colleges offer semester-long courses or weekend workshops; 

° Other public or private organizations, user groups, computer 
stores, manufacturers, and vendors offer occasional or regular 
workshops. ' , » • 

When instructional objectives are clear, and inservice needs assessed, the 

4 N ° 

committee can investigate and evaluate software, and finally determine what 
hardware is required (Swalm, 1983). 

A contingency approach is better than no-planning or lockstep plan- 
ning. And we approve of Fisher's (1983) admonition to leave lots of space 
in whatever plans are developed: :, A good plan will provide time for 
schools and teachers to 'get up speed,' to become informed and trained in 
computer use so they can make effective .decisions; it will also leave room 
for serendipity and individual differences." (Fisher, 1983, p. 13.) 
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THE COST OF INSTRUCTIONAL' INFORMATION SYSTEMS: 
Results From Two Study Districts' 

James S. Catterall 
University of California, Los Angeles 

Research Interest In school district pupil assessment and In the ties 
between this activity and Instructional practice had progressed with a pre- 
dictable logic over the past few y6ars. A nutshell hi story "oT^Waluation 
Inquiry would suggest that long-standing efforts to understand and Improve 
the art and craft of evaluation" have made room recently for Investigations 
of just how the results of evaluations are used by educators In the 
schools. And a consequence of this new focus has been a curiosity, prima- 
rily academic thus far, In the costs and benefits of performing and utiliz- 
ing evaluation (1). The Evaluation Systems Pruject at UCLA's Center for 
the Study of Evaluation reflects this evolution of attention In Its current 
research program, and reported here are the findings of a second major 
Investigation 1n the area of "evaluation costs" by Center research staff. 

This report accompanies the work of Richard C. Williams and Adrlanne 
Bajnk on school district Instructional Information systems. Their recently 
coined title refers to formally-linked testing/pupil information/Instruc- 
tional planning systems evident In some elementary and secondary school 
districts (2). The research describes several models of systems fitting 
such a description; two of these, through case studies, were exam1ned^1n 
depth during the past year. Oyr .Interest centered on two types of 
analyses: describing such systems and exploring ways of gauging their 
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impacts (reported by Williams and Bank elsewhere) (3), and- identifying and 
probing issues related to the costs of such systems. This second effort is 
reported here. 

It may surprise few readers that what we learned in the process of 
pursuing this cost investigation may be of more lasting importance than the 
cost findings themselves. So we report on both sorts of results. First 
the two district systems are briefly described, and a rationale for their 
selection for this study offered. Then the cost data and analyses are 
presented, along with a comparison of these findings to some related 
results from our previous testing costs research (4). Finally, we describe 
certain lessons emerging from our efforts — lessons concerning what 
researchers may expect when approaching field inquiries into educational 
program costs, and lessons regarding the nature of the policy issues 
imbedded in the costs of evaluation systems. 

• Study Models: I. Student Achievement and 
II. School Improvement 

Model I 

The student achievement model describes a system developed over the 
past dozen years in f a small California school district as part of its 
curricular emphasis on individualized instruction. In this district, > 
teachers use the results cf twice-annual, criterion-referenced achievement 
testing in order to place students in classes, to group youngsters within 
classes for instructional purposes, to assess the effectiveness of their 
curricular strategies, to prescribe remedial activities when needed, and to 
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provide a basis of communication with parents. While alV of these purposes 
are commonly attached to school district assessment practices of one type 
of another, the study district illustrating this model incorporates these 
objectives into a tightly linked "system/ Not only are these purposes 
served by a single battery of tests, appropriately geared to grade levels, 
but also the district's core instructional continua in reading, .mathema- 
tics, and language skills have been developed in. tandem with the tests by 
the district's teachers and staff. The instructional program and the 
assessment instruments are thus intentionally matched, and the information 
generated by the assessments is viewed commonly by district personnel as 
both relevant and salutary for instructional planning and improvement. 

Pupils are tested in the fall and winter of each school year. Results 
of tests, scored and elaborately organized through district data processing 
services, are available to teachers within a week (5). Learning special- 
ists at each of the district's schools assist in test administration and 
interpretation of results. Principals use the test results as the primary 
basis of fall and spring planning sessions with individual teachers — a 
critical component of the district's instructional leadership activity. 

The student achievement model is. now an ongoing, stable, and dominant 
fact of the district's instructional life. Both daily instructional acti- 
vities and incidental assessments of pupil progress are directly geared to 
the scope and sequence of topics outlined in the continua. Teachers and 
administrators universally reported the central ity of this "system" to us 
in our investigation. 
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Model II 

The school improvement model was named for the California educational 
reform initiative which spawned its creation (5). Legislation over the 
past ten years has created statewide a pattern of school site planning and 
programming, which includes the provision of discretionary money to schools 
for the purposes of carrying out activities identified at each school as 
critical to its improvement. This planning is performed by school site 
councils, staffed mainly by teachers and parents at each participating 
school; planning for instructional emphasis is thus decentralized. In- 
structional decisions are based partly on test data and partly on the 
results of systematic surveys of various school constituents. A norm- 
referenced test (The Iowa Test of Basic skills) is administered annually 
each spring to all pupils in grades 2 through 6. In addition, parents, 
teachers, and some pupils are surveyed to probe their perceptions of 
strength and weakness in current instructional programs. 

. The results of the tests and surveys are used. by the site councils as 
as a basis for allocating discretionary resources made available to each 
school due to its participation in the state program — funds in excess of 

$100 per year per pupil. These resources are commonly used for specialized 

r — : -■ f—. - - - .*. 

instructional materials, or for hiring aides to assist in critical areas of 

the instructional program. The overall thrust of planning efforts under 

this model is to assess pupil progress in key areas of the curriculum, and 

to guide future instructional efforts toward areas of greatest perceived 

need. Our study district illustrating this model employs it is 40 of its 

44 elementary schools district-wide. 
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The two study districts were chosen according to straightforward cri- 
teria. In previous inquiries by our staff into district evaluation prac- 
tices, these two districts had exhibited strong evidence of having a co- 
herent "system" of pupil assessment and instructional administration. 
Since the heart of our proposed work was an exploration of attributes and 
costs of such systems, we were willing to consider any that appeared to be 
using a tightly linked system. And we were encouraged by central staffers 
in both of these districts through their willingness to provide information 
and access to us, and further by their willingness to broach questions of 
cost in the analysis. As we discuss below, this latter concession may be 
more difficult to obtain in district settings than might be first 
supposed. 

Approaching System Costs \^ 

\ 

The costs of a particular program or distinct set of activities in an 

. \ \ 

organization can be thought of in a number byways. Three., approaches domi- 
nate the literature surrounding cost analysis: Budgetary costs, resource 

• ' ■ \ J 

or ingredient costs, and opportunity costs (7K\By budgetary costs we mean 

\ ■ 

explicit expenditures directed toward an activity'and identified in an 
organization's budget statements. As might be expected, only when consi- 
dering very distinct enterprises is this approach to cost analysis very 
meaningful. For example, if a school district budgets-for and operates a 
drop-in center for its out-of-schopl youngsters — replete with its own 
staff, facility, and so forth — its written budget might reflect a close 
approximation of the monetary cost of such a program to the district. In 
this case, analysts might look to the district's budget documents for 
program cost estimates. 

* x - 100 
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For our purposes, and for those of most program cost analyses 
generally, the budget is very unrevealing. Most school district programs 
and activities are supported. by the efforts of teachers, administrators, 
staff, parents, and pupils who allocate their time toward a multitude of 
ends, often simultaneously. And just how much effort lies behind any one 
pursuit is not often formally accounted-for. Our task of evaluating the 
costs of pupil information systems is a case in point. Both of the systems 
under our scrutiny are supported in part by all of the participants noced 
above, each devoting varying amounts of time. An understanding of the 
costs of maintaining these two models must necessarily entail knowing 
something about the extent and value of the time of the people involved, as 
well as the value of other material resources devoted to the systems. A 
resource or "ingredients" approach involves just this, focus, as does the 
analysis presented below. 

But before proceeding, a final question remains which has led to a 
third dominant construct in the analysis of costs: how should the costs of 
resources associated with an activity or program be valued? One suggestion 
is that market value approximations be drawn for each. Thus, for example, 
the value of a day of teacher time could be°estimated to equal the 
teacher's annual compensatipn divided by total days worked per year, admin- 
istrator time could be similarly valued* costs of facilities used could be 
annualized and prorated to their various users, and so on. Trorn a deci- 
sion-making standpoint, however, a more important "cost" that; a school 
district bears in devoting Its resources to a particular activity is that 
it, cannot devote such tied-up resources to something else. That resources 
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have varying values to their owners in alternative uses has led to the idea 
of "opportunity costs," defined to represent the value of resources to a 
decision maker in their best alternative deployment. 

This analysis approaches the cost:; of maintaining Models I and II from 
an ingredients perspective — our central interest is to identify the full 
range of resources attached to each system in the study districts, and to 
generate estimates of the monetary value of these resources. We could thus 
compare "system" resources to total pupil expenditures in our desire to 
gauge their overall importance. The analysis presented acknowledges the 
importance of the "opportunity cost" perspective, but for reasons cited 
concentrates on identifying resources and esti mating their market values. 

We relied heavily on the chief coordinator of each of the two instruc- 
tional information systems studied in order to generate resource esti- 
mates- Both were intimately familiar with their systems. Where they did 
not know how much time or who was involved in performing, certain activi- 
ties, or how much was expended for purchased materials or services, other 
district personnel were consulted or appropriate district records checked. 
This approach seems to have led us to a relatively complete picture of each 
model one that withstood crosschecks with various district personnel. 
The possibilities of. bias in responses offered to us are discussed below. 

We organized our search for system resources, and report, the results 
Jiere, at three levels of district operation as well as for the district 
overall. For both Models I and II, we identified activities at the central 
office, school, and classroom levels. Totals of resources were recorded in 
their primary units (such as hours per week or fraction of full time, or 
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actual dollars spent). For purposes of analysis, we calculate monetary 
equivalents for these resource allocations using estimates of district 
salary scales. In addition, per-pupil calculations are provided to 
facilitate comparisons. 
Costs of Model I, Student Achievement 

The resources needed to maintain the CRT testing and instructional 
management system of 'ffodel I are displayed in Tables 1 through 4. Table 1 
presents the costs at the central district office level. These consist 
primarily of partial time allocations of central staff, and to a lesser 
extent the costs of test scoring services and material purchases for such 
things as answer sheets. The coordinator spends on average a little less 
than one day per week in support of the system over a typical year. In 
addition, an assistant superintendent reported spending about a day per 
month on system activities, as did the district instructional materials 
coordinator. A little more than one third of a secretary's time at the 
central level is required, primarily to assist with the processing of test 
results and with the generation of reports used by teachers and 
principals. 

These time allocations have been valued in the table according to 
approximate salary levels and fractions of time devoted to the system. 
TfieseTfactors r are" shown in-the-tableT — The-overaTI— level-of-central 

resources shown, about $22,000, amounts to a little more than $4.00 per 

-> 

pupil in the district. 
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f , T able 1 

Model I: Central District Costs 



Type of cost Cost Estimate 

A. Personnel 

° Evaluation Coordinator 

(17.5% FTE @ $34,000 $ 5,960 

° Instructional Materials Coordinator 

{5% FTE @ $30,000) $ 1,500 

° Assistant Superintendent 

(5% FTE @ $40,000) $ 2,000 

° Clerical Support 

(37.5% FTE @ $18,000) $- 6,750 

Total Personnel Costs $16,210 

B. Equipment and Materials 
? Computer 

(17.5% devoted. to CRT, annualized cost of ($10,000) $ 1,750 

° Paper and Materials 

- Answer sheets $ 750 
~ - Photo copying $ 750 

- Printing • - $ 2,500 

Total Equipment and Materials Costs $ 5,750 

C. Total Central District Costs $21,960 

D. Per Pupil Cost $ 4.22 ~ 



Costs at the school site level, based on the same approach, are 
greater than those at the central office level. This is primarily because 



of the significant "amount of Time £pehtT)y"t^ 

ning instruction on the basis of system reports. Each principal spends a 
full week twice pen year in one-to-one consultations with teachers to 
assist in instructional management. For each principal, this contributes 
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to a total of more than a ten percent allocation of time on a yearly basis 
to the system. For each teacher, this planning activity occupies about six 
hours per year. In addition, a learning specialist at each school site 
devotes one day every other week to system activities. These school site- 
level costs, shown in Table 2, amount to a total of about $15.00 per pupil 
over the year. 

Table 2 

Model 1: Site Level Costs (non-testing) 

Type of cost Cost Estimate 

A. Principal 

(2 weeks plus 1/4 to 1/2 day 

per week ongoing = 12% FTE @ $30,000) $ 3,571.00 

B. Learning Specialist 

(10% FTE @ $28,000) $ 2,800.00 

C. Media Specalist 

\2% FTE @ $25,000) $ 500.00 

D. Teachers 

(6 hrs. @ $17 for each of 22) , $ 2,244.00 

E. Total Cost $ 9,115.00 

F. Per Pupil Cost $ 15.19 

The remaining costs, those for testing, of conducting the Model I 
system are displayed in Table 3. In addition to; spending 5 to 10 hours per 
semester in administration of the tests, teachers spend about 5 hours in 
preparation and grouping youngsters for testing, and some teachers receive 
brief inservice sessions related to the testing program. In addition, some 
items on the tests are teacher- or aide-scored, and the values of these 
time allocations are shown in the table. The time pupils spend taking 
tests and the time of parent volunteers have been recorded in the table, 
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but no dollar approximations have been made. Testing costs identified 
amount overall to a little more than $17.00 per pupil. 

Table 3 

Modal I: Testing Costs (per 30 pupils) 

Type of cost Cost E sti mate 

A. Pre-Test Activities 

0 Teacher planning: 5 hrs. (@ $17 per hr.) $ 85.00 

0 Teacher inservice: 1/2 hr. (not all * 

teachers each year) $ 8.50 

B. Test Administration 

0 Teacher: 15 hrs. per year average 

(5 to 10 hrs. per semester) $ 255.00 

° Pupils: 15 hrs. per year $ n.b. 

C. Scoring and Analysis 

° Teacher-scored items: 8 hrs. per year average $ 136.00 

° Aide: 4 hrs. @ $10 per hr.^ ' " $ 40. GO 

o 

0 Parent Volunteers: 4 days $ n.b. 

Total Testing Costs (30 pupils) $ 524.50 

Testing Costs Per Pupil $ 17.48 

D. Total District Testing Costs (3800 pupils) $60,600.00 

*n.b. = Non-Budget Item 

The costs of Model I are summarized in Table 4. Central office, 
school site, and pupil testing costs total about $34.00 per pupil in the 
district. To this figure we might add a factor representing the value of 
pupil time involved for testing (about 15 hours per year) to achieve an 
overall picture of resources supporting the Model I system. 
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Table 4 

Model I: Total System Costs 



Type of cost 
Central Costs 



Cost Estimate 



A. 



o 



Personnel 

Equipment and Materials 



$ 16,210.00 
$ 5,750.00 

$ 21,960.00 



o 



total 



B. 



C. 



School Site Level (non-testing) 
° Coordination and development 

School Site Testing (524.50 per 30 pupils) 



$ 63,805.00 
$ 60,600.00 
$146,365.00 



Total 



D. 



Total Costs Per Pupil 



$ 



34.00 



The Costs of Model II: School Improvement 

As described earlier, there are substantial differences between the 
Student Achievement Model and the School Improvement Model . as systems for 
guiding instruction. Whereas the chief activities of the first are to test 
pupils, analyze test-generated information, and modify teaching activities, 
the primary methods of the second are to conduct a comprehensive planning 
process and to direct specific added resources to identified areas of 
instructional priority on the basis of planning outcomes. As we might 
expect, observed patterns of resource use and costs for the two systems 
differ as well. The costs of Model II are presented in a manner parallel 
to^the discussion above. Tables 5 through 8 present district, school site, 
and classroom level costs as well as a summary of costs respectively. 

The central district costs for Model II were more plainly evident both 
to researchers and district staff than were those for Model I. Two profes- 
sional staff members — an evaluation specialist and a resource teacher — 
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devote themselves entirely to the Model II planning and evaluation system. 
Other central office personnel involved inc'ude a secretary at about half- 
time, and a small amount of temporary clerical assistance. Costs for these 
personnel (with total benefits to these staff shown separately) are dis- v 
played in Table 5. The table also shows costs for contracted research ser- 
vices (for assistance with the constituent surveys), and for word process- 
ing and printing. The total central costs, a little less than $100,000, 
amount to between $6.00 and $7.00 per pupil. 

Table 5 
Model II; Central Costs 



Type of cost 

A. Personnel 

° Evaluation Specialist 

° Resource Teacher 

° Secretary 

° Temporary Clerical 

° Benefits 



B. Contracted Research Services 

C. Word Hrocessing Services 
° ' Equipment Rental 

° Maintenance Contract 
° Supplies 



D. Printing 

E. Miscellaneous 

F. Total Central Office Costs 

G. Costs Per District Pupil* 



Total 



\ 

Total 



Cost Estimate 

$27,600 
$18,800 
$ 8,000 
$ 1,500 
$15,500 

$68,300 
$16,000 

$ 2,500 
$ 1,000 
$ 1,500 

$ 5,Q00 

$ 7,000 

$ 1,000 
$97,300 

$ 6.41 



15,178 pupils in grades 2-6 
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The costs of Model 1 1. incurred at school sites, not including those 
for testing which are discussed separately below, are concentrated prima- 
rily in the time of various individual s devoted to planning and manage- 
ment. The school site councils engage teachers and parent volunteers in 
planning for constituent surveys and in making instruction-related recom- 
mendations based on the results of surveys and pupil tests,. Administrators 
assist site councils in ongoing monitoring of instructional activities at 
each school. The amounts of time spent on the system for each of these 
types of people are recorded in, Table 6. For teachers and administrators, 
dollar approximations of these costs are presented. Again, volunteer time 
is noted but not translated to dollar equivalents. The total site level 
costs amount to almost $9.00 per pupil. 

Table 6 

Model II; Site-level Costs, Non-Testing 

Type of cost - Cost Estimate 

A. Planning 

° Administrator 

- (1 day @ $150) $ 150.00 

° Certificated Staff 

(1 day & $100 for each 16) $1,600.00 

° Community Volunteers 

(1 day for each of 12) $ rub,* 

Total Planning Costs $1,750.00 

B. Ongoing Program Management 
° Administrator 

(1 day per month over 9 months) SI, 350. 00 

° Certificated Staff 

(1 hr. per month over 8 months for 

each of 16 @ $17 per hr.) ■ ; $2,176.00 

° Community Volunteers n.b.* 

i - Total Program Management Costs $3,526.00 



C. Total Site Level Costs Per Site $5,276.00 
). Costs Per Pupil 
*n.b. denotes non-budget costs 



D. Costs Per Pupil $ 8.80 

* 
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Testing costs for Model II are shown in Table 7. The annual adminis- 
tration of the test (The Iowa Test of Basic Skills) requires three hours of 
teacher and pupil time, as well as two hours of a reading coordinator's 
time per classroom. Tests are administered to all pupils in grades 2 
through 6. The costs of this testing amount to a little less than $3.00 
per pupil per year. 

Table 7 

Model II: Site-level Testing Costs Per 30 Pupils 

Type of cost Cost Estimate 

A. Teachers 

(3 hrs. 0 $17) $ 51.00' 

B. Reading Coordinator 

(2 hrs. 0 $17). $ 34.00 

C. Pupils - 

(3 hrs. each) $ n.b.* 

Total Cost $ 85.00 



D. Total Site Level Testing Costs (grades 2-6) $42,955.00 

E. Per Pupil Cost $ 2.83 
*n.b. - Non-Budget Item 

Comparison of Models I and II 

Table 8 presents summaries of the various monetary costs discussed in 
the previous section. The comparative figures illustrate fundamental dif- 
ferences between the two models. The most obvious difference is in the 
amount of testing time devoted to support each system. Model I's CRT 
assessment occupies youngsters for about 15 hours per year and their 
teachers for even longer.. In contrast, the ITBS administered for Model II 
is completed in 3 hours. This results investing costs of more than $17 

no 
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per pupil in the one format and less than $3 per pupil on the other. The 
second major difference in the two approaches, at least in their generation 
of costs, is the relative abundance of teacher and principal time required 
by Model I for instructional management activities. This is shown in Table 
8 where site-level non-testing .costs are more than $12 per pupil for Model 
I and less than $9 per pupil for Model II. The central -office costs of 
maintaining the two models are about $4 and $6 per pupil, the more expen- 
sive being Model II where full time staff are allocated to the pupil 
information system. The overall scorecard indicates that in estimated 
dollars per pupil, Model I is considerably more expensive to administer ~ 
$34 versus $18 per pupil. 

One additional perspective is generated in Table 8. When district 
operational expenditures per pupil are considered, each model requires less 
than 1 percent of district spending for its maintenance — the more expen- 
sive Model I taking up about 1 percent and Model II requiring about 2/3 of 
1 percetit of district per-pupil -expenditures. This overall level of system 
resources devoted to instructional information "systems is consistent with 
the findings of our previous research into the costs of all pupil achieve- 
ment testing' conducted in school districts. Here we found that testing for 
all purposes, of which the type of testing discussed here is a subset, 
accounted for approximately 3 percent of district expenditures (8). 
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Table 8 

Model I and Model II: Cost Comparisons 



Location 
of Cost 



Model I 



Model II 



Total 



Central Of-fice $ 21,960 

Site Level ,• 

non-test $ 63,805 

Testing at Site $ 60,600 

Total Costs $146, 365 5 

Percent of P.P. E. / 

--— V-,--- 1 ! " 



Per Pupil 
$ 4.22 

$ 12.27 ■ 
$ 17.48 — 

$ 34.00 

• 0.97%2 



Total 
$ 97,300 



$133,570 



$ 42,955 
$273, 825 4 



Per Pupil 1 
$ 6.41 . 

$ 8.80 
$ 



$ 18.04 
0.67% 3 



1) 15,178 pupils, grades 2-6 

2) Estimated $3,523 p.p.e.** 

3) Estimated-$2, 700- p.p.e.. 

4) Plus community volunteer time and pupil time 

5) Plus parent volunteer time and pupil time 

*p.p.e. denotes estimated per pupil expenditures for district operations 

Implications for Deci-sionmaking? 

The view afforded by this analysis of the costs of instructional 
information" systems suggests that these enterprises are , not particularly 
•costly, arid that they are not potential ^purges of real! ocatabl e r esources 
for districts looking for cost-saving changes. Ten to thirty dollar per 
pupil stakes in the face of $3,000 annual per pupil expenditures are not 
cause for immediate alarm. This conclusion 1s reinforced by the limited 
degree to which even these small costs could be considered discretionary. 
If the costs of the two models are explored for direct expenditures which 
could be curtailed by decisions to not spend money -(such as for supplies or 
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purchased services) few candidates emerge. Table 9 shows that of Model I 
costs, only about 2 percent fall Into this category, and for Model II, 
/discretionary costs amount to less than 11 percent. 

Table 9 

Model I and Model II: Direct (discretionary) vs. Indirect Costs 



Model I 

Total System Costs - $176,661 

Discretionary Costs . $ 4,000 
(Paper, photocopying, 
printing) 



Model H 

. $273,825 

$ 29,000 
(Word processing, miscella- 
neous, contract "research, 



Percent discretionary 
costs. 



2.3% 




. The notion of opportunity costs further informs our discus\ion at this 
point. While the various displays in this report generally show small. dol- 
lar approximations of the many resources contributing to the pupil Infor- 
mation systems studied, they also portray these resources in ways tha^t 
facilitate this alternative conception of costs. To illustrate, if th^e 

evaluation director spends all of his time on one of our "systems," he\ is 
, ■ ' ' * \ 

unavailable for other pursuits. If a school principal spends 2 weeks on 

one required task (such as in conferences, with teachers in Model. I) , he. 

cannot spend this time on something else. In short; any % time devoted to^an 

instructional information system in a school "district, or to anything for \ 

that matter, has an "opportunity cost." That cost is the value of what 

that time might gain for the 'district if spent differently. 
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The most valid approximation of the magnitude of the "opportunity 
costs" implied by any chosen activity is the value of the best alternative 
use to which the resources taken-up might be applied. The opportunity 
costs of the systems studied here are represented in this analysis only 
indirectly. The amounts of resources devoted to the systemsy most signifi- 
cantly the time commitments of teachers and administrators, are listed,.- but 
no attempt is made to assess the value of the opportunity costs involved. 
Estimating just what these professionals might accomplish with their time, 
toward similar or unrelated ends, must submit to further analysis or at 
least to the cogitations of those contemplating resource use decisions In 
light of this information. We do not perform these exercises as a part of 
this report. But such analysis could contribute to answering some impor- 
tant questions, such as: -What sorts of instructional information systems 
are most cost effective? ~ under what circumstances? Are they worth 
supporting at allr 
Other Lessons 

We promised earlier that we learned more than just cost estimates by 
pursuing this research. We attempt to catalogue these observations here: 
H Cost inquiries require creative detective work on the part of re- 
searchers. Program costs are not customarily, recorded inaccessible form 
in school d^irlct documents. This is due primarily to the multiple con- 
tributions of the primary district actors — teachers, administrators, and 
pupils — to an ample range "of programs e and goals of the district's 
schools. This is also due to a historical lack of district sponsored cost 
analyses similar to these attempted here. 

114 
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2) Accounting^for. the ingredient or resource costs of a particular, 
school program requires that researchers develop a complete picture of the 
functioning of the program under scrutiny. The basic questions of this 
accounting « who and what are involved and to what extent — demand such 
understanding. As such, remaining members of our team of researchers, who 
wished to focus on system characteristics or estimates of system impacts, 
also benefitted from the activities of the cost analysts. In a converse 
sense, researchers providing descriptive analyses of programs may have less 
additional work than they might first suppose if they wish to extend their 
analyses to areas of cost. 

3) Since program cost analyses of the sort described here ave rarely 
undertaken within school districts, the subjects of proposed research such 
as this may evidence either disinterest. or reluctance when solicited for 
participation. Disinterest may be caused by the impression that the the 
information would have little practical utility and therefore not warrant 
any costs of cooperation. Additional skepticism may becaused by, the 
simple fact that cost analysis is presumed to be driven by a quest for 
economies. A potential result of a cost analysis is the" suggested reallo- 
cation of resources, and if a program is portrayed by researchers or inter- 
preted by decision makers to be expensive, it may suffer pressure for 

'budget reductions. Since the type of knowledge generated by cost analysis 
is not generally available for a full range of most school district pro- 
grams, the subjects of proposed research may feel singled-out and threat- 
ened by a proposal. This has been overcome in our research so far through 
our guarantees of confidentiality, through a shared exploratory curiosity 
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among researchers and subjects, and probably through the convictions of our 
respondents that the systems under study in fact do not command extra- 
ordinary resources, 

4) Some bias may be anticipated in the responses of school personnel 
to questions regarding the resource demands of their programs. We relied 
in our research on the expertise of the sponsors and administrators of the 
instructional information systems studied. Their self-interests may be 
presumed to lie in casting their programs in a positive (i.e., least expen- 
sive) light. (See #3 above.) We did not uncover instances of underesti- 
mation in this research. Much of what was told to us by sponsors was veri- 
fied by the views of teachers and other district personnel, but we did not 
systematically verify all information that contributed to our estimates. 
For this reason, the estimates reported should be considered reliable and 
"lower-boundary" in nature. 

5) A key to understanding the meaning of these cost findings is the 
development of further .knowledge of alternative ways of providing the sort 
of instructional information systems studied here. We have looked at only 
two models in detail, and the elaboration^ a full range of actual or 
potential configurations of such .systems would assist researchers inter- 
ested in questions of efficiency -- a natural extension of our work thus 
far. 

•v . 
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- LIMKINGJTESILNG_VIITH INSTRUCTIONAL DECISION MAKING: 

SOME MODELS AND GUIDELINESFROM RESEARCH 

by ; ' 

Don Dorr-Bremme 

Introduction 

The nations investment in school achievement testing is enormous; the 
amount and variety of testing have continued to grow. Unfortunately, how- 
ever, much of this testing remains under-utilized. Achievement-test re- 
sults can be extremely useful in school-wide and district-wide planning and 
decision making. They can provide information for analyzing curriculum, 
diagnosing instructional strengths and weaknesses, and determining direc- 
tions for school improvement. Test-score patterns can also serve as the 
basis for projecting student achievement goals and monitoring progress 
toward their attainment. But, as four years of research at UCLA's Center 

for the Study of Evaluation shows, it is only the rare district in which 

c>- ...... 

schools regularly and systematically use testing for these or similar pur- 
poses. In most school districts across the country, testing and instruc- 
tional decision making are not routinely and coherently linked. As a re- 
sult, few schools and districts are getting a maximum return on the dol- 
lars, staff time, and student learning time that they invest in testing. 
At the same time, the quality of the educational planning in their schools 
suffers from a lack of of precise and useful Information. 

What distinguishes the relatively small number of schools in which 
principals and teachers regularly pay close attention to test scores? How 
do some districts structure on-going links between testing and instruction 
in their schools? How can others do so, achieving both fuller use of test- 
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ing and more informed instructional decision making in the process? This 
article addresses each of these questions, drawing answers from the Center 
for the Study of Evaluation's recent, extensive research. 

After a brief review of the research base, the discussion below opens 
with a look at current test-use patterns in the nation's schools. These 
patterns verify that much achievement testing is indeed under-utilized, and 
they suggest that there is little use of test information in systematic, 
school-wide planning. The data also indicate that leadership by district 
administrators is a key factor in how closely testing is linked to instruc- 
tional decision making and reveal two general approaches that district 
leaders can follow in building such links. Attention then turns to some 
specific ways that each of these general approaches can be pursued by dis- 
trict leaders. Models and concrete guidelines are presented, all based 
upon intensive case studies of districts that have systems in place for 
linking assessment with instructional planning and decision making. A 
summary highlights the main features of these systems'. 
The Research 

From 1979 through the present, the Center for the Study of Evaluation 
(CSE) at UCLA has been conducting two, complementary studies of testing and 
test use. One, the Test Use in Schools Project, has studied current 
nation-wide patterns. Inquiry has focused on many types of tests and other 
assessment techniques, especially in the basic skills areas of reading/ 
English and mathematics. The project's central effort was a 1981 survey of 
assessment practices, test uses, and contextual factors that influence 

them. Questionnaires were! sent to the principal and four classroom 

i i 
i 

>m 119 



113 



teachers in elementary schools and high schools randomly chosen from within 



received from 1,058 respondents: 220 principals, 475 upper-elementary- 
grade teachers, and 363 high school teachers of English and mathematics. 
Fieldwork took place before and after this survey; a total of eleven 
schools (four secondary, seven elementary from five districts in different 
geographic regions of the country were visited. The two phases of field 
study produced over 100 in-depth interviews with school- and district-level 
educators, including 12 principals, 69 classroom teachers, various instruc- 
tional specialists, ar o. er administrative personnel. 

While the Test Use in Schools Project has focused on what is, the 
second CSE effort, the Evaluation Design Project, has yielded a detailed 
look at what can be. It has conducted fieldwork 1n districts which cur- 
rently have testing-Instruction linkage systems in place.* In order to 
locate these districts, recommendations were widely sought and between 40 
' and 50 of the districts nominated were screened In phone Interviews with 
appropriate district-level admlnistators. Ultimately, eight districts were 
selected for Intensive exploration 1n on-site Interviews and observations. 
A first phase of the Evaluation Design 'district, organization study {1979- 
1982) was directed at understanding how the test1ng-1nstruct1on linkages in 
these districts worked and at Identifying the district characteristics that 
contributed to. their creation and maintenance. More recent work has 
examined the costs and impacts of two of these eight systems, each of which, 
represents one ge.:3ral approach to tying test results with Instructional 
decision making. \ 



a nationally representative sample of school districts. Returns were 
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While the Test Use Project establishes the need for linking testing 
with instruction and indicates some general approaches for doing so, study 
of these eight districts by the Evaluation Systems Project provides some 
clear guidelines for effective testing-instruction linkage systems. 
The Status Quo.: Two Tiers of Testing and Limited Use 

The Test Use in Schools Project provides a portrait of the 
contemporary status quo. It describes central tendencies in public-school 
test use across the nation. This portrait reveals that there are two tiers 
or layers of student-achievement assessments in our schools today. These 
are consistently distinguishable from one another in their priori etorship, 
characteristics, and functions. One tier of assessment is internal or 
local to the schools. It is "owned," and for the most part produced, by 
teachers themselves. This local or internal tier includes two main. types 
of assessment: (1) the tests, quizzes, and other measures that teachers 
construct and administer in the course of their teaching, and (2) the cli- 
nical judgments of students 1 achievement that teachers form as they inter- 
act with students and observe their work in various classroom situations 
day after day. A third kind of measure also figures in this tier, but it 
is especially imporant for elementary-school teachers. These are the tests 

C ■ ■ ■ 

included with comnercial curriculum materials used .in the classroom. While 
these are not produced in the school, teachers in the elementary grades are 
most often invested in them. Teachers often have a say in choosing (and^ 
. c h o o s i rig ^ e r s — 

can time their administration and adapt 'their content to fit the pace and 
emphases of instruction. 
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The second tier of assessment is external to the school: mandated by 

. i ' 

the district, state, and/or suggested by federal program requirements 
(e.g., for placement* in compensatory education programs). Norm-referenced, 
standardized test batteries are the most common among these. Other types 
of measures used for minimum competency (or functional literacy) testing or 
as part of state assessment program? are also included here. In some 
cases, too, tests constructed or purchased by districts and referenced to 
their curricular objectives fall in this second category. Tests of these 
kinds are also included there. In some cases, too, tests constructed or 
purchased by districts and referenced to their curricular objectives fall 
in this second category. Tests of these kinds are developed beyond the 
schools. Their administration is called for primarily to meet organiza- 
tional needs and concerns at higher levels of public-education governance. 
Those who work at those levels may have a sense of ownership in these 
tests; educators in the schools rarely do. ^ 

These two tiers of assessment function quite differently in most 
schools and districts. Teachers and principals rely heavily on the results 
of internal assessment strategies and consider them important as they go 
about routine instructional planning and decision making. At the same 
time, they generally treat information from external testing as of minor 
importance, using it only occasionally and idiosyncratically. These 
patterns are obvious in both CSE's fieldwork findings and survey data. 

Forty-three classroom teachers^ were interviewed during pre-survey 
fieldwork in a systematic-but-open-ended format. They discussed all the 
information they had throughout the year on students 1 academic capabili- 
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ties, performance, and progress; they described whether and how they used 
-that— information^ — Collectively, they- cited far more- uses- for the informa- 
tion that came from assessment strategies that were local to the school and 
classroom. (See Table 1.) \ 

Teachers surveyed across the nation were asked to rate the importance 
of diverse types of assessment results in four routine, decision-making 
tasks. Again, the pre-eminence of the internal tier of assessment was 
apparent. (See Table 2.) Principals in CSE's national survey were asked 
to, rate hew important a role data from various sources playecfTp^eight 
regular school-level administrative activities. Here, the separate 
functions of the two tiers of achievement assessment was especially 
apparent. Principals reported counting internal assessment data more 
heavily in making instructional ly relevant decisions, e.g., allocating 
funds, assigning students, evaluating teachers. But- they indicated that 
results of external measures were more important in reporting to those 
beyond the -school-, e.g., to district administrators and the public. 

Taken together,, the research findings just cited show that there are 
notable; quantitative differences in the ways the external and internal 
tiers of assessment are used by educators in the schools. They reveal that 
the results of externally mandated testing serve fewer purposes (Table 1) 
and-are not counted as heavily in planning or decision making (Tables 2 and 
3). But fieldwork clearly suggests that there are^a-lso significant qua! 1- 

i * 

tative differences in how the two tiers of assessment are typically uti- 
lized by teachers and principals. The results of external tests a. < 
often examined -briefly, casually, and asystematically. Do principals 
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Table 1 

Types of Tests and the Uses of Their Results by Teachers (Interview Data) 

(Cells show the number of times the 44 interviewed teachers freely cited each use 
for each type of test) 



TEST TYPES 



USES 





A 


B 


C 


D 


E 


F 


G 


H 


I 


Total 


Planning Instruction 


24 


21 


10 


3 


2 


3 


13 


4 


2 


82 


Referral /Placement 


3 


6 


0 


2 


0 


0 


11 


1 


0 


23 


Within Classroom Group- 
ing & Individual 
Placement 


6 


14 


18 


4 


6 


5 


4 


3 


1 


61 


Holding Students 
Accountable for Work, 
Discipline- 


8 


2 


3 


0 


0 


0 


0 


0 


0 


13 


Assigning Grades 


32 


8 


17 


5 


1 


1 


1 


1 


0 


66 


Monitoring Students 1 
Progress 


18 


12 


17 


2 


1 


1 


0 


1 


0 


51 


Counseling & Guiding 
Students 


10 


6 


0 


0 


1 


2 


3 


0 


0 


22 


Informing Parents 


0 


1 


0 


0 


0 


1 


0 


0 


0 


2 


Reporting to District 
Officials, School 
Board, etc. 


0 


3 


1 


0 


0 


2 


0 


0 


0 


6 


Comparing Groups of 
Students, Schools, etc. 


0" 


1 


0 


0 


0 


1 


1 


0 


0 


3 






















Certifying Minimum 
Competency 


0 


;0 


0 


0 


0 


0 


0 


1 


0 


1 


TOTAL USE CITATIONS 


101 


74 


63 


16 


11 


19 


33 


10 


3 


330- 


Explicit Statements 


0 


1 


0 


1 


0 


0 


10 


2 


7 


21 



of Non-use 
KEY: 



A = Teacher Constructed X 
B = Teachers 1 Other Major Assignments 
C = Curriculum Embedded j 
D = School /Department/Grade Level ! 
E = Comnercial Diagnostic 



F = District-Objectives Based 
G = Standardized 
H 3 Minimum Competency 
I = Statewide Assessment 
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Table 2 

Importance of Test Results for Teacher Decision-Making 
in Elementary and Secondary Schools (Survey Data) 



III Ul CMMIVUI J wws*wii*-vi. j — — — " 

Trnean ratings~on 4-point scale! S^CruTralT^-^Iniportant; 
2 = Somewhat Important; 1 = Unimportant or not used] 



ELEMENTARY 



Decision Area : 

Planning teaching at 
beginning of the 
school year 

Initial grouping or 
Placement of students 

Changing a student from 
one group or curriculum 
to another, providing 
remedial or accelerated 
work 

Deciding on report card 
grades 



A • B . C D 

3.39 2.53 

3.12 3.58 2.91 2.51 

3.12 ^ 3.66 .3.04 2.52 



3.38 3.69 2.89 1.62 



E 

2.60 

2.59 
2.52 



1.81 



SECONDARY 



ERIC 



Planning teaching at 
the beginning of the 
school year 

Initial grouping or 
placement of students 

Changing students from 
one group or curriculum 
to another, providing 
remedial or accelerated 
work 

Deciding on report card 
grades . 



3.59 



3.04 3.84 2.48 



3.27 3.61 2.67 



3.65 3.68 2.29 



A -'Teacher-Made Tests 

B * Teacher Observations/Opinions 

C a Tests Included with Curriculum 

D « Standardized test batteries 

E a District Continuum or Minimum Competency Tests 



■ u 



Q X 
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2.22 

2.28 
2.52 



1.36 



2.38 

is 

2.46 
2.59 



1.45 
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consider the results of standardized and district-objectives-based tests in 
curriculum evaluation? Table 3 suggests that they do. But interviews 

Tndi cate t!Tat^hTs~ofT^^^ — tfre~sco res^ 

mention them iTTa""f acuity meetingVand "^Tnt~6W^the"~aTeas~~i nwhi ch the 
school did especially well or poorly. Do teachers use standardized test 
results in planning? Apparently they do to some extent (Tables 1 and 2). 
Fieldwork suggests, however that, more often than not, this means a once-" 
a-year visit to the office for quick look at their students' cumulative 
files. Are standardized test batteries and minimum competency scores con- 
sulted in student placement? "'Each table indicates that they are. But 
visits to schools make clear that they are most often consulted as part of 
an automatic or cursory gate-keeping procedure. Law or policy guidelines 
direct that students with scores below a certain cut-off point be ^placed in 
a compensatory program or remedial class. Alternatively, as one high- 
school teacher put it, describing a procedure reported by many offices: 

They give me each kid's standardi zed-test score on njy class 
roster. If one stands out, I usually check with the 
counselor to be sure the kid should really be assigned' to 
geometry. 

Such uses contrast sharply with teachers' recurrent and systematic use of 
assessment that are local to the classroom and school in an on-going pro- 
cess of intructional planning and decision making. They contrast markedly 
with principals' serious consideration of teachers' advice, recommenda- 
tions,^and grades on teachers assignments in making budgetary decisions or 
next year's class assignments. And they certainly do not constitute 
thorough utilization of external testing data in a systematic process of 
school -wide analysis of curriculum and instruction, decision-making and 
planning. 
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Table 3 

Importance of Test Results for Principal or Schoolwlde Decision-Making 
in Elementary and Secondary Schools (Survey Data) 

[mean ratings on 4-point scale: 4 = Crucial; 3 » Important; 
2 = Somewhat Important; 1 = Unimportant or not used] 



-EtEMENTARY- 



Decision Area : 
Curriculum Evaluation 



Student Class Assignments 



Teacher Evaluation 



Allocating Funds 



Student Promotion 



Public Communication 



Communicating to Parents 
Reporting to District 



2.94 
(.84) 

2.93 
(.79) 

2.12 
(.97) 



3.05 
(.70) 

2.31 
(1.05) 

3.43 
(.55) 

2.62 
(.91) 



B 

3.27 
(.64) 

3.12 
(.71) 



3.08 
(.71) 

3.29 
(.67) 



3.45 
(.57) 



3.01 
(.67) 

2.50 
(.81) 

1.70 
(.76) 

1.91 
(.87) 

2.65 
(.81) 

2.77 
(.90) 

2.91 
(.60) 

3.12 
(.68) 



2.91 
(.75) 

2.35 
(.91) 

1.53 
(.78) 

1.89 
(.90) 

2.31 
(.96) 

2.47 
(.99) 

2.64 
(.98) 

2.78 
(1.10) 



3.04 
(.87) 

2.46 
(.99) 

1.80 
( .93) 

1.94 
(1.01) 

2.38 
(.94) 

2.34 
(1.00) 

2.67 
(.95) 

2.74 
(1.10) 



2.99 



2.44 



1.68 



1.91 



2.45 



2.52 



2.74 



2.88 



SECONDARY 



Curriculum Evaluation 
Student Class Assignments 
Teacher Evaluation 
Allocating Funds 
Student Promotion 
Public Communication 

Communicating to Parents 

« ■ . 

Reporting to District 



2.76 
(.75) 

2.98 
(.73) 

2.39 
(.83) 



3.33 
(.85) 

2.24 
(1.05) 

3.56 
(.55) 

2.53 
(.88) 



3.14 
(.70) 

2.99 
(.79) 



3.34 
(.54) 

3.46 
(.75) 



3.38 
(.76) 



2.83 
(.67) 

2.77 
(.77) 

1.63 
(.74) 

1.73 
(.81) 

1.61 
(.78) 

2.84 
(.80) 

2.91 
(.S3) 

3.10 
(.64) 



3.27 
(.64) 

2.98* 
(.87) 

1.77 
(.71) 

2.20 
U.13) 

2.58 
(1.28) 

2.92 , 
(1.03) 

3.03 
(1.00) 

3.12 
(.97) 



2.95 
(.82) 

2.78 
(.87) 

1.84 
(.78) 

2.06 
(1.08) 

2.05 
(1.13) 

2.30 
(1.07) 

2.55 
(.99) 

2.92 
(.95) 
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A ■ Results of Teacher and Curriculum tests 

B ■ Teacher Opinions/Recommendations 

D » Standardized, norm-referenced test batteries 

E » Minimum Competency Tests 127 

F " District Objective-based or Continuum Tests 

6 3 Average External . Tests (D,E,F) 



3.02 



2,84 



1.75 



2.00 



2.08 



2.69 



2.83 



3.04 
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Why do the two tiers of achievement assessment 'function in the dif- 
ferent ways that they commonly do? The reasons are not hard to find. They 
lie in the interplay of several factor s: characteristics of the measures 
themselves, circumstances surrounding their availability, educators 1 train- 
ing in assessment and the organization of educational planning in schools, 
districts, and beyond. Examining these factors not only discloses why 
internal and external assessments are used differently in the schools. It 
also begins to reveal how each tier of assessment can be used more fully 
toward Improving instruction — and why district leadership 1s the key to 
this process. 

Our system of schooling is organized such that teachers routinely do a 
great deal of instructional planning. They have a major role in planning 
what to teach (and/or emphasize) and how to teach 1t, 1n diagnosing Indivi- 
dual students 1 learning needs, and 1n assuring that students are working at 
appropriate levels 1n the curriculum. As the school year unfolds, they 
need to monitor their students 1 progress, to consider whether and how to 
adjust the pace and emphases of their teaching, to grade students and In- 
form parents of achievement-to-date, and so on. To do all this and do 1t 
well, teachers, need assessment tools wth three basic characteristics: (1) 
Validity — they must assess what the teacher believes he or she has actu- 
ally taught 1n a way that seems consonant with the way he or she has taught 
1t; (2) Suitability — their Intended purposes must fit the tasks the 
teacher needs to accomplsh, (thus teachers seek placement tests for place- 
ment, chapter and unit tests for monitoring progress and grading, etc.); 
and (3) Immediate Availability — the teacher must be able to employ them 



122 



whenever it seems appropriate to do so and have the results back promptly. 
In short, the assessment tools that teachers need must be sensitive to 
^Ixrcal^ondTttbns— to-^^ 
cular classrooms at the moment. And, in order to function throughout the 
year as the instructional leaders of their schools, principals need 
measures of the same kind. It is not surprising, then, that both teachers 
and principals rely heavily on assessment strategies that are internal to 
the school and its classrooms; teacher-made tests, and assignments, 
teachers 1 observations and clinical judgments, and the adaptable, readily 
available tests that come with the comnercial curriculum^ materials they are 
using. From their points of view, these internal measures have all three 
of the characteristics listed above. Externally mandated measures, on the 
other hand, usually do not. They are not designed primarily to provide 
data for routine classroom decision making. The fit between their contents 
and format and a particular teacher's curriculum is problematic. Often, 
their scores are not returned until weeks or months after administration. 
Often too* the results come back in a format teachers and many principals 
find unfamiliar and/or cumbersome. For any or all these reasons, the re- 
sults of standardized tests, other minimum-competency measures, and many 
di strict-objectives-based tests can seem remote and irrelevant to teachers 
and principals. In addition, teachers and principals generally have 
limited formal training in testing and measurement or the use of test data 
(Herman and Dorr-Bremme, 1983; Yeh, Herman, & Rudner, 1978). This limits 
also the accessibility of external testing data /to educators in the 
schools. CSE's Test Use Project field found teacher and pricnipals voicing 
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these very concerns as drawbacks of external testing (Burry, et al., 1982; 
Dorr-Bremme, in press). 

But the very characteristics that make internal assessment tools ideal 

-for-u-seH^H^d-fv4dua i t- j tea<^er-s^nd-p^inaipaU 

restrict thejr utility for systematic school- and district-wide planning . 
Their content and the timing of their administration 1s idiosyncratic, 
variable from classroom to classroom. Aggregating the data they provide in 
order to see achievement patterns across grade levels, a department or the 
entire school, therefore, is difficult if not inappropriate and 
impossible. This is especially true of teacher-made tests and assignments, 
but it also often applies to tests embedded 1ri texts and other commercial 
materials. (Teachers time their administration differently; they sometimes 
adapt their contents. The same materials or text series are not always 
used throughout the school.) And while teachers 1 cumulative observations 
and experience-based judgments are valuable sources of Information, they 
cannot be readily synthesized into a precise, detailed, picture of specific 
curricular or teaching strengths and weaknesses across many classrooms or 
school $. 

It 1s these problems with local or internal assessment strategies that 
have made standardized, minimum-competency, and special district-object- 
ives-based tests attractive to local school districts '— and make similar 
measures a virtual necessity for states and other educational agencies. By 
providing standard and consistent data across settings, such tests facili- 
tate comparisons among classrooms, schools, and/or districts; they permit 
year-to-year monitoring of performance. They are likely to be more sound 
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psychometrically than teachers' own tests; in most circumstances they are 
sufficiently valid to indicate broad patterns and trends. Tests of these 
kinds can take time to administer, score and analyze comprehensively, but 
c omprehenstvene^ — 
if data are gathered only annually or bi annually. Coming full circle, how-, 
ever, the same features that make these types of measures useful to 
districts and larger education agencies generally limit their usefulness 
for teachers and 'principals. Thus, two tiers of achievement testing, 
largely distinct in their functions, are maintained in public schooling. 

Both of these tiers could be used more fully than they now are as 
information sources in school-wide decision making and planning for 
instructional improvement.* How? The data and analysis presented above 
suggest two approaches that districts can follow: One approaches to build 
from the inside out : to'construct district tests that have the character- 
istics of internal assessment tools — the validity for local curricula, 
suitability for routine classroom purposes, and inmed late avail ablity that 
appeal to teachers — and at the same time provide consistent, re*i1abla 
data that can be aggregated in ways useful for school and district decision 
making. The second approach 1s to build f^m the outside in : to analyze 
information from externally mandated measures currently given in the dis- 
trict and deliver it to schools at times and in formats that maximize its 
utility in planning for curricula;* and Instructional Improvement. 

These approaches are not mutually exclusive; both can be followed 
simultaneously. But the effectiveness of either depends upon more than the 
proper handling of testing and test scores. It also depends upon a 



district system that structures and. supports the use of testing information 
in an on-going planning process ~ systems of a type that are not widely 
present 1n most districts today. On the whole, most districts do not rou- 
tinely return test results to schools in ways that facilitate their use in 
decision making. Admnlstrators review scores for the faculty 1n most 
schools, but rarely on a periodic basis as part of routine procedures. 

I? 

Follow-up to assure that teachers are giving attention to the content area, 
skills, etc., that test scores indicate need emphasis 1s rarely routine 
either. (See fable 4.) Survey data show that the majority of teachers are 
Instructed 1n how to administer tests and that they are Informed about test, 
results. Yet 1t appears that few receive training 1n how to link teaching 
and testing or 1n how to use test results in Improving instruction. (See 
Table 5.) These are only some very general -Indicators that not many dis- 
tricts are closing the test1ng-1nstruct1on loop with systematic planning 
mechanisms. They are supported, however, by *fi el dwork from both CSE pro- 
jects. Furthermore, even though efforts of the kinds shown 1n Tables 4 and 
5 are only the most elemental. In a district testlng-tnstructlonal decision 
making linkage system, they can make a difference 1n how teachers view and 
use testing. Analyses of survey data show that where there 1s more support 
by district and school leaders y for the use of test results in planning, and 
where there 1s more staff development 1n assessment, teachers have a signi- 
ficantly more positive view of testing and its uses and tend to treat, the 
results of d1str1ct-object1ves-based, standardized, and even minimum-compe- 
tency tests "as more important in Instructional decision making. (Table to 
be Included 1n final draft.) With this 1n mind, let's examine some ways 
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Table 4 

Making and Holding Teachers Accountable for Test-score-Based Curricular Decisions (Survey Data) 



,SCHOOLADMINISIRAIOR(S). , . , 



Principals' Reports* Teachers' Reports* 
Elementary Secondary Elementary Secondary 



. Meets ulth teachers to review scores and , 3,09 2.94 284 2 05 

identifies areas that need ^xtra emphasis , 

Observes teachers, reviews their plans 3,23 ^ 2 .66 2 ,31 

to ensure areas Indicated by tests are \ 
being emphasized < 

Takes test scores Into account In evaluating 1,57 1,55 i,j 
teachers and/or establishes test-score goals 
for teachers to meet 



DISTRICT ADMINISTRATOR^ . , , . , 

Returns test results such that they can be 2,63 2,03 Not Asked • 

used In school's curricular decision making 

Observes, reviews school plans and/or 2,84 2,67 " 

requires reports to assure school is 
emphasizing skills that test 
: show need work 

Establishes specific test-score goals for school 2,12 2,33 ■ 

133 ; 

Mean ratings on four-point scale : 4 « happens regularly, routinely; 3 * not regular or routine but happens fairly often; 
V 8 not regular of routine and happens rarely; 1 ■ does not happen at all, . 



Table 5 

Percentages of Teachers Reporting Recent Participation in Staff Development 



Secondary Secondary 

Topic Elementary Engl i sh Math 

41) — How to administer -tests-required iy ™1. ± 

ny state, district, and/or school 78 54 46 
(procedures to follow, etc.) 

(2) Analysis and explanation of state* * A 

district, or school test 'results - 84 70 60 



(3) How to tie what is taught more closely 
to the skills, content covered on 

required tests 50 * 37 „ 25 

(4) Training 1n the use of test results ■ 

to improve instruction 35 21 19 
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that districts can create successful links between testing and planning for 
instructional improvement in their schools. 
Building Links From the Inside Out 

Districts that follow this approach build outward from classroom 
assessment needs to those of the school and districts. They also build 
from what should be taught to .what should tfe tested. First they construct 
district curricula, then district tests to match. 

Two of the districts studied closely by CSE's projects were especially 
successful in taking this approach. Their slightly different testing- 
instruction linkage systems are useful models for others. 

a 

The Central City Model* 

Located in the rural midwest, Central City School District serves 
about 5,000 students in seven elementary schools, three junior highs, and a 
high school. It has a long history of innovation and commitment to curri- 
culum development. It also has a group of teachers who pioneered use of 
the high school's main-frame computers (originally purchased and used for 
computer-assisted Instruction) 1n the $Por1no and analysis of teacher-made 
tests. These factors, and an energetic leader, joined in the creation of 
Central City's system for linking test information with instructional 
planning. 

District names used in this paper are pseudonyms • Any resemblance 
between these names and those of actual districts and communities is 
unintended. 
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The test information . Each summer in recent years, -the district has 
sponsored curriculum development projects. But while the district initi- 
ated, compensated, and guided, itfwas teachers who did thfework. Several 
representatives from the faculties of each school were selected by their 
peers to participate. 

Efforts began with the construction of an elementary-grade media (or 
library) skills module and continued through the development of complete 
mathematics and social science curricula for the elementary grades. Latere 
the mathematics curriculum was extended through grade 8 and work began on a 
reading program. In each case, development was done unit by unit in 
several stages. First, teachers decided on instructional objectives and 
-seVected^and/or^wote-mateH 

them. Then, pre-and post-tests referenced to the objectives of each 
unit were designed and "mastery levels" for each objective were specified. 
Units and accompanying tests were piloted the next year; objectives, mate- 
rials, % and test items were revised in light of teachers' criticisms and 
suggestions. Further revisions incorporating teachers 1 feedback were made 
after the units went into general use in schools across the district. 

Testing materials were designed such that all the unit tests could be 
scored and analyzed by computer and returned to the teachers in a day or 
two. Results came in the form of a set of easy-to-read sheets, one for x£;- 
each student. The sheet, listed each objective covered on the test, the 
number of items that measured the particular objective, the number of these 
items the student had correct and incorrect, and whether. the number correct 
equaled "mastery." At the top each sheet appeared a paragraph that 
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described the types of errors the student had made and summarized the types 

M.^i ff 1 culM.es_.th 

covered. 

In mathematics, the district had selected a sample of items from the 
unit tests and combined these to' create mid-year and end-of-the-year sum- 
mary measures given to students in all schools. Teachers received summary 
sheets of the type described above for these tests, too. (The district was 
considering developing similar tests in other subject areas once the pro- 
cess of curriculum and test-item revision was considered complete.) 

All this applies to the lower grades, but similar developments had 
begun in the high school mathematics department. These were initiated by 



t required, to* follow 



the teachers, who had worked toward common currlculja^and devising 
computer-scored tests for various courses. In linef with a general district 
attitude, other departments were encouraged, but n< 
this example. L 

The end results of the district-wide effort (> were several: (1) curri- 
cula that were consistent across the district, that teachers were invested 
in, and which teachers actually used; (2) a system of tests, that fit the 
curricula and provided timely information in a form appropriate for a vari- 
ety of routine Instructional decisions; and (3) a body of test Information 
that was valid and consistent from classroom to classroom. and could thus be 
aggregated and compared in school and district planning. 

The structure of school decision making . ■ Within the schools, these 

test data came into play In two main ways. .First, they were routinely used 

' ■'/ • ' \. \ 

by teams of teachers in regular "unit" meetings. Elementary-school "units? 

138 1 
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included several teachers (one of whom was chosen as unit leader) a clus- 
ter of students across two or three grades, and occasionally an Instruc- 
tional aide. , Students were often divided among % un1t teachers in different 
groupings for different. subjects based oh their current level of achieve- 
ment and rate of learning. (Some schools, however, tended to use the 
self-contained classroom approach for most students). 

Unit teams met at least weekly during release time at the end of an 
abbreviated school day. At the beginning of the yeari" they discussed 
students' placement and planned Instructional emphases and pacing. Later 
on, they routinely examined students' progress, reviewed their placements, 
re-evaluated and altered their teaching,^ and" discussed individual learner's 
problems and how best to address themTT'llat as "welTT" 

as other available Information, were routinely examined as these matters 
were considered. Unit meetings, then, were the primary setting for Unking 
test data with instructional decision making. (Where classrooms were self- 
contained, teachers reported using the district tests Individually, as well 
as in unit meetings. ; And similar procedures were followed in the junior 
high and high school math departments.) 

A second use of district test data occurred periodically as principals 
established school goals and agendas for school In-service activities. 

District support systems . The linkage effort described above was 
supported by the Central School District in a number of ways. 

First, district leaders Initiated and provided resources for the 
currfcul um-and-test development. They also gave release time for weekly 
unit meetings in which test data are used for instructional planning. 
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Second, district administrative leaders provided staff development in 
curri cuTum-wri ting and -test-developent. — Qrigina]ly-,_thes.e^^ 
weekly "courses" were led by professors from a state university. Later, 
however, the district encouraged teachers to take over the classes: to 
adapt them to be more practical and relevant and to serve as Instructors.,: 
Credit on the district's pay scale was given for participation 1n these 
classes. A district administrator also maintained close contact with the 
nearby office of the local Intermediate Educational Agency (IEA). IEA help 
was routinely sought in problems in test development, and scorlhg-anti- 
analysls Issues. The IEA also provided some staff development in 
instruction. 



Third, the district maintained media centers staffed by Instructional 
specialists 1n each school. Specialists helped unit teams and Individual 
teachers locate supplementary teaching materials to address learners' 
needs. They also offered training 1n such areas as Instructional diagnosis 
and prescription. 

Fourth, a district administrator worked with teacher committees 1n 
piloting curriculum units and tests, eliciting teachers' critiques, and 
revising objectives, materials, and test Items. 

It was this same administrator who encouraged continuing and broaden- 
ing the use of the computer-scor1ng-and- test-analysis process. 

The Shelter Grove Model 

The Shelter Grove Unified School District 1s located 1n the 
southwestern region of the country. Until three years ago, Shelter Grove 
was an elementary school district. The recent merger has brought Shelter 
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Grove's enrollment to about 5,700. These students are distributed through 
four elementary sphools, two middle schools (grades 6-8) , and' a four-year 
high school. 

Shelter Grove's system for linking testing with instruction is similar 
to Central City's in several ways. Yet it is different enough to be worth 
descr-iptio^^ ■ 

The test information . Like Central City, Shelter Grove administers 
tests of several types; But those that have the greatest power to influ- 
ence instruction in Shelter Grove schools are those developed by the dis- 
trict and referenced to its continua (or sequences) of instructional objec- 
tives in reading, mathematics and writing (composition). 

Shelter Grove'iniTiVlly contracted"^ th « a cornmerci ai firm whi^ti pro^~ 
m1sed to write test items for district-selected objectives and to provide 
computer printouts of scores. Introduced in the early, 1970' s, these tests 
failed to win teacher support. Teachers complained that they were not 
coordinated with anything that was taught, that they did not know what to 
do with the results. 

Teacher committeesTwere appofnl^Ht^t^ tesf^Ttefiist Tfiey~~ 

responded to the need for/coordinating testing, and curriculum by beginning 
to work on a district-lfcvel continuum of objectives. From then on Shelter 
Grove's experience paralleled the more recent history of Central City. By 
the late 1970's, teacher conrnittees had devised continua of objectives and 
accompanying criterion-referenced tests for reading and math, as well as 
similar tests for language arts. More recently, a district writing con- 
tinuum was established. v 
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Unlike the, Central City materials, Shelter Grove's tests do not serve 
as unit pre-tests or post-tests. And except in written- composition, dis- 
trict objectives are not accompanied by district-designed material s or 
recommended learning activities. Rather, the continua are aligned with 
commercial reading and math text series used district-wide. 

The district. tests at the appropriate level were routinely adminis- 
tered to students by classroom teachers at two or three points between 
October and February. Scores were aggregated by the district's Testing 
Coordinator for individual students, instructional groups, entire classes, 
and the school. These, profiles were sent to the schools, 1n time for plan- 
ning days that occur regularly at several points through the year. 

In addition, proficiency tests composed of various sepents <>f the 
district's criterion-referenced . tests were administered to children 1n~ 
grades 4, 5 and 6 each year 1n April and May 1n accordance with, state 
requirements. 

The structure of school decision making . District tests were rou- 
tinely used 1n each ele ment ary and middle school during planning days that 

occur at several points 1n the school year.' (The system has yet to-be 

introduced 1n the district's high school.) Two of these days were 1n 

June. 0 On the first, the program of the school was routinely evaluated by 

.-»'"■ V 
the entire school staff looking at the group, classroom, and .total school 

scores. These sessions functioned as a needs assessment for the next 

. " ^ _. .7 „..:.,._ , * ' 

school year. On the second June planning day, Individual teachers plaped 
students in appropriate learning groups for the coming year using the 
test-result profiles on each student. 

• • ■■- 

- ' . 142 . 

o .. - ■ ■ • •• ' - 

ERIC 



135 



In September of each year, test information was updated; information 
on students new to the district was added. In October*, teachers met with 
their principals to set learning goals — benchmarks oft the continuum that, 
based upon past performance profiles, they expected the children in each 
, Instructional group to meet. s 

A mid-year evaluation took place each February. Summary reports on 

current-year t^TlTT^^ere'riDiT^dl stri butedr~and-examined^ — Principal s-met 

with teachers, as well as with the Superintendent and Assistant Superinten- 
dent for Instruction, to discuss students 1 progress. Plans for modifying 
the Instructional program were made at, this time. Then, 1n June, the cycle 
began anew with reference to the again-updated test-score profiles. 

Individual teachers also used criterion-referenced test Information 1n 

reporting to parents'each October and again each spring. Report cards 

■> - 

listed continuum skills on one side and noted students 1 progress toward 
" each objective. And each May, letters were sent to the parents of children 
who were two grade levels behind expected performance; special conferences 
with these parents were also arranged. ■ 

District support systems . As was the case 1n Central City, a number - tt 
of district activities and programs helped to sustain the Unking of test 
data with Instructional planning 1n Shelter Grove. In addition to the dis- 
trict's leadership and resources 1n developing the Instructional -objectives 
contlnuua and criterion-referenced tests, these included the following. 

The district maintained a Professional. Development Program (PDP) that 
provided teachers with the skills necessary to act upon the test results. 
Coordinated by a full-time specialist, the PDP had evolved over time based 
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upon the Madeline Hunter orientation to teaching. Level One activities 
(for all new teachers, aides, and substitutes) dealt with such basic teach- 
ing skills as understanding goals and objectives, motivation and reinforce- 
ment, and task analysis and diagnosis. Level, Two activities (which were 
not required but encouraged, and which many teachers took) extended those 
of Level One with emphasis on individualizing instruction. Strategies for 
meeting~affective"needs using inquiry skills, and teaching specific curri- 
culum .content were also covered. (Prior to the general implementation of 
this PDP program, all principals had been required to take the Level One 
course plus courses in clinical teacher supervision.) 

PDP sessions require teachers to apply the skills taught back in their 
own classroom, with supervision and feedback from the PDP coordinator. 

Learning specialists conducted demonstration lessons, recommended 
materials, conducted diagnoses of new students, and assisted teachers 1n 
planning and placement when new criterion-referenced test scores arrived 1n 
the schools. The learning specialists 'were considered master teachers, and 
regularly played an Important role in helping teachers use test Informa- 
tio n. T hey also functioned to communicate changes in the continuum or 
changes 1n district policy to the faculty. With the PDP, learning special- 
ists were perceived as critical supports to the district's linkage effort. 

A Testing Advisory Committee composed of a principal and several 
teachers continually updated and Improved the district's tests in light of 
teacher criticisms. This group also handled whatever administrative and 
technical problems arose 1n testing, scoring, and reporting back results* 
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Ad hoc continuum revision committees made/up of teachers and learning 

ii 

specialists were paid during the summer to revise sections of the continua 
as seemed appropriate. 

In* addition to these formal organizational features",' TV^rTe^ty'bf 
other networking activities (e.g., principal observations, learning specia- 
lists 1 visits to classrooms, monthly meetings of a district communications 

pa 

council) helped district personnel work closely together 1n maintaining 
links between test data and Instructional planning 1n the Shelter Grove 
schools. r \. \ 

Guidelines 

The experiences of Central City^ and Shelter Grove, especially 1n 
contrast^to-those^ofM:w^ 

linkage systems, suggest a number of guidelines for other districts 
Interested 1n Unking testing with Instruction from Inside out. 

1. Build curriculum and assessment measures together "in-house . 11 

Administrators and teaching staff 1n both districts believed very 
strongly 1n the district development process. They felt that 1t helped 
assure teacher "ownership" and confidence 1n both curricula and tests; 
these, in turn, seemed important as prerequisites to teacher use. Shelter 
Grove's unhappy experience with tests built outside the district, even when 
they were coordinated to district specifications, supports this wisdom. 

2. Assure a close fit between test Items and currlcular objectives and 
materials . - ~~ — — — — - • ' . - .• • • 

This can best be done by designing curriculum first and then the 

tests, as was done 1n Central City and, ultimately, 1n Shelter Grove too. 
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Teachers are inclined to see district objectives-based or criterion- 
referenced tests as a burdensome irrelevancy 1f this condition 1s not met. 
New Branford, an urban district with 30,000 enrollment 1n the northeastern 
United States, attempted to c criterion-referenced tests keyed to Its 
district reading and math objectives. But when Test Use Project 
researchers visited New Branford schools, they found that few teachers used 
them. Continuum objectives were Intended to fit with the five or six math 
and reading series used across the district. In fact, according to 
teachers, they fit well with none. Thus, teachers continued to use the 
tests Included with these conmercial series to get the Information on 
achievement they needed and they also had to give district tests to 
comply with district requirements. But Information from the latter was 
rarely consul ted , *andT;Taclrer3^ 

sons, Central City teachers neglected their district's objectives-based 
reading tests, although they were generally enthusiastic about those 1n the 
other subjects. Developed years earlier with little teacher participation, 
and no accompanying curriculum materials, teachers complained that the 
reading tests were not valid for the content of the two basal series used 
1n Central City. 

3. Strive for maximum teacher Involvement . . 

To help build curriculum and tests that teachers own snd use, 
teachers' participation 1n the development process must be more than nomi- 
nal . Both Shelter Grove and Central City Included many teachers on their 
development committees; these teachers did the real work of constructing, 
the curricula (or contlnua) and the test Items. Mechanisms were provided 
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that allowed all district teachers to offer feedback on a regular basis. 
Their criticisms were taken seriously in the revision process. 

In contrast, New Branford (mentioned just above) and Metro District 
(another urban district studied by the CSE Test Use Project) had a small 
number of teachers on district advisory committees as they constructed 
contlnua of objectives and accompanying tests. These teachers did not par- 
ticipate in the actual development process; their presence was not visible 
to dlstrict^Faculty; they had l rEtTe~Tmpa^™^ 

district did teachers feel the objectives or tests were completely suit- 
able. New Branford teachers' response has beendescribed. Teachers' 
response to Metro District's tests was quite mixed. 

4. Make tests that coyer the entire range of skills in the curriculum 
and/or continuum of. objectives . ■ : " 

The district tests of Central City and Shelter Grove included Items 

that v assessed students' performance on skills and content from the most 

elemental to the most advanced in the subject areas tested. Metro District 

(enrollment over 100,000), 1n contrast, purchased tests for each grade 

level In reading, math, and language arts that covered only the most simple 

skills to be taught 1n the grade. In the economically disadvantaged neigh- 

borhoods^where-more-students-had-trouble-w1th-these skills, -test-results 

did help teachers locate areas in which individuals and class groups needed 

remediation. But in these- school s, the tests also functioned to push the 

actual curriculum 1n the direction of the most elemental skills. Teachers 

•and principals wanted students (and their schools) to do well on the tests 

each spring. Thus, they spent much time drilling and re-drill 1ng children 

on the simple skills tested. Simultaneously, they gave shorter shrift in 
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their teaching to other skill? -secified for the grade level, which were 
included c the t*rt. Jsewrrsre ir ~he district, where students routinely 
'oh* j ? > er , u 100 per rrect on the same tests, the tests 

yieiued Vi ccle v., „ jnostic or r , .jcu.ont information for teachers. 

One moral of these contrasting stories, then, is test what you want 
teachers to teach, because teachers will place their teaching emphasis on 
what you test. - : t . 

—Several other— 'doV- and "don 1 ts" can be abstracted from the Central 

City, Shelter Grove, and similar but less successful models. These, how- 
ever, are equally pertinent to the "outside in" linkage approach discussed 
next. Thus, they will be omitted here and mentioned in the concluding 
summary. 

Building Links From the Outside In 

Districts that follow this approach adapt Information from externally 
mandated tests to suit schools 1 planning needs. In so doing, they support - 
school -level planning structures and procedures, just as districts taking 
the inside out path do. 

The testing-Instruction linkage systems of two districts that followed 
the outside 1n approach are described below* They provide: very different, 
but equally Instructive models. 

The St. John Model 

The St. John School District covers a wide geographic area of suburban 

and semi -rural municipalities In a Western state. Its 72 schools serve 

•■ » ■ ■ 

between 40 and 50 thousand students 1n grades K^12. 

• . vt? .. 148 ■ 
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Linking testing with instructional planning began /in St. John during 
the mid-1970 1 s when the state legislature enacted a program intended to 
stimulate planning for school improvement at local school sites. Participa- 
tion in the program was voluntary, but over the years most St. John elemen- 
tary schools, along with two of its junior high schools and one high 
school, elected to participate. The district encouraged this involvement; 
in turn, schools 1 participation occasioned district efforts to provide test 
data for use in local site planning. 

The test information . Long before the advent of the state-sponsored 
school improvement program, St. John School District had required adminis- 
tration of the- Iowa Test of Basic. Skills. Students were tested each 
January-+n grades 2-6. The purposes this information had served previously 
are not germane here. But once numerous St. John schools joined the state 
program, test data became especially important for them. Guidelines for 
the state school -improvement- planning process required that in establishing 
improvement plans schools specify: (1) the "existing level of performance" 
i n a-particul ar. area , (2 ) the "needed program changes or_ additions," (3) 
improvement objectives, and (4) activities to measure these 'objectives. 
Major, activities to be undertaken in order to achieve the objectives also 
had to be described, along with budgets and other Improvement program fea- 
turesi But the four requirements enumerated here were those that caVled 
for "hard data" such as test results. 

It seemed reasonable to use ITBS results in developing these improve- 
ment plans, yet district administrators realized that these came back from 
the test publisher in a form that was cumbersome. Computer printouts 
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presented the results for each sub-test area for each grade for each year 
on a separate page. Principals and teachers found these reports compli- 
cated as well as overwhelming in size. Consequently, the district under- 
took development of what it now calls the Academic Performance Profile 
(APP). 

In summary, the APP gave each district elementary school an annual 
overview of its ITBS test results for all years and all grades for a par- 
ticular subtest (e.g., reading comprehension, .math concepts, etc.) on a 
single page. This reduces fifty pages of computer printout to approxi- 
mately six ordinary 8 1 /2 by 11 inch pages. 

o 

In addition, the APP simplified the format in which the information 
appeared. Simple graphs were devised to 0 visually display : (1) the scores 
of student groups as they moved through the grades (1982 first graders as 
second graders in 1983, etc.); (2) the performance at various grade: levels 
in various years (the fourth grade in 1981, 1982, 1983, etc.); and (3) the 
gains (indicated in terms of grade-level growth) realized from one year to 
the next for the various grade levels (the gains made from second grade in 
1982 to third grade in 1983). Two simple tables on each page (i.e., for 
each sub-test) supplemented the three line graphs. 

Since the state program guidelines also call for annual needs as$£ss- 
ment, the St. John District also created survey questionnaires for staff, 
parents, and students. These solicited respondents' perceptions of the° 
effectiveness of the schools 1 various programs, as well as their percep- 
tions of how strong the need is to improve in the same areas. Each school 
can add up to 20 questions to those common acrossthe district. Surveys 
are administered annually in the spring of each year. The. district's 

*0 t • ' . ■ :o ■ 
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evaluation office tabulates those for each school and returns their results 
in a concise form. 

The structure of school decision making . The state's school. Improve- 
ment program mandated the creation of a School Planning Council (SPC) 1n 
each participating school. SPC members had to include the principal and 
representatives of the teachers, other school staff, parents and other 
community members, and (at the secondary level ) students. _Th Is group was 
assigned 'central responsibility for establ 1shing r needs, goals, and activi- 
ties for- school improvement, as well as allocating the state program funds 
provided to' the school for Improvement activities. 

However, St. John's district evaluation specialists elaborated on 
these state requirements. They urged their schools to also create 
"component committees, 11 smaller groups (including SPC members and others) 
who were charged, with planning for improvement in particular o areas — 1n 
subject areas, in school environment, in human relation, in staff develop- 
ment, etc. 

Component committees review the ITBS/APP summary forms, survey re- 
sults, and other information. They specify and document needs, set objec- 
tives, and develop school and classroom activities to realize them. They 
also state how achievement of the objectives will be evaluated and propose 
a budget suitable for their plan. In a next step, various component com- 
mittees present their particular plans to the School Planning Council. The 
SPC accepts or suggests changes in each Improvement-plan component and 
makes decisions regarding final allocation of state program dollars among 
the various components. The SPC also monitors implementation of the plan 
through the coming school year. 
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While plans were routinely developed for a three-year period, revi- - 

<i ■ 

sions were made each spring based on information gathered during the cur- 
rent school year. Thus, school Improvement planning was an annua.l process 
centered 1n the spring. But Implementation 1n .classrooms and SPC monitor- 
ing recurred throughout each school year. 

Interviews with participants and observation of planning meetings 
indicated thai test data (and survey results) were used in deciding upon 
and substantiating needs, specifying objectives, evaluating Implementation, 
and revising the plans. SPC members also referred to this Information 1n 
making and justifying budgetary decisions. 

District support systems . The St. John School District supported Its 
test1ng-1nstruct1on linkage system in many of the same ways that Shelter 
Grove and Central City supported their quite different ones. 

Staff development in the organization and process of planning, Includ- 
ing the use of the APP test summaries, was conducted for 600 district per- 
sonnel during their first year 1n the state program. Others received this 
Introductory training as they entered the program. Furthermore, teachers, 
principals, and parents agreed that the regular availability of the dis- 
tricts 1 two evaluation specialists was a key to the program's maintenance. 
They routinely provided staff development and answered ad hoc questions 
regarding planning and test-data use. 

St. Jotin also maintained a comprehensive staff -development program 1n 
Instructional techniques. This was a major factor 1n facilitating the 
realization of school plans. . 

The Bayview Model 

Bayvlew 1s a community of 100,000, and is located about 50 miles from 
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a major Western metropolitan area. The Bayview Unified School District's 
sixteen elementary schools, four junior highs, and three senior highs 
enroll 14,000 students. 

Bayview' s si x-year-old effort at testing-instructional linkage was 

-more diffuse than that in most of the other school districts visited by CSE 
researchers. Interest in Resting and evaluation was relatively new, and 

-many-i n-the _di str-lct_were- skepti cal of thei r val ue . Norfethel ess t JftfilSeed- 
to comply with externally mandated testing programs stimulated a small 
group of district administrators to try to make greater local use of them. 
Only one of these uses will be discussed ^here. It offers an. example of . 
"outside in" testing-instruction linkage that is quite different from the! 
St. John School District's. 

. The test information . Ihffee different achievement testing programs 
figured in the Bayview linkage endeavor to be described here. The first of 
these was the State Assessment Program (SAP). This half-hour test was 
administered each spring to students in grades, 3, 6, and 10 in accord with 
state requirements. The test was devised by the state and referenced to 
objectives common to many state-approved text series. Items were matrix 
sampled; not every student was asked to respond to Identical questions. 
Thus, data for individual students were not reported. Results focused on 
grade level and school patterns. 

A second test used by Bayview was the norm-referenced, standardized 
Comprehensive Test of Basic Skills (CTBS)* The district had just begun to 
require this test in all schools for grades 1-9 when CSE fieldwork was 
conducted. Formerly, 1t had been given 'Only 1n schaols-wlth Title I (now 
Chapter 1) compensatory education programs. 



146 

The district's proficiency (or minimum competency) testing program was 
also used in testing-instruction linkage. Forms for grades 5, 9, 10, and 
11 had been developed with the help of consultants to meet the state's . 
mandate. These measures covered reading, writing, and mathematics skills 
deemed essential for life coping. The current forms of the test were 

introduced in 1978. 

— i * 

The decision-making structure . The data from these three tests was 
brought to bear on instructional planning in several ways by Bay view dis- 
trict leaders. Chiefly, however, they had begun to use the three test pro- 
grams mentioned above as content for staff development course work in task 
analysis and diagnostic-prescriptive teaching. . • - 

District leaders had won grant funds from the state to create a 
professional Development Center (PDC). The primary , focus of the PDC's 
program was the continuing development of effective teaching strategies. A 
Teacher Center funded by a federal grant augmented the PDC. Curriculum 
development and the translation of educational research for practical, 
instructional applications were the central thrusts of the Teacher Center's 
program. The very presence of these two centers testified to Bayview.'s 
emphasis on teaching effectiveness skills. In addition, principals were 
required to attend workshops dealing with supervision, and these focused on 

' J: } 

the elements of effective teaching. 

It was in the context of increasing external test mandates and the 
emphasis of staff development that Bayview's linkage system began to take 
shape. From the perspective of District leaders, Bayview teachers and 
principals were avoiding facing the issues raised by the District's 
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relatively poor performance on the external measures* In response, said the 
Director of Staff Development: 

We [at, the central office] tried to model a problem-solving 
/ way of looking at It, so principals could do similarly 1n 
/ their schools* The Director of Instruction worked with 
/ principals 1n the way he wanted them to work with teachers. 
/ Also, we asked teachers 1f they were addressing areas of the 
/ test* They said they were. When we observed, -we found 
• J teachers had difficulty defining the skills to- be' taught. <as 
well as diagnosing for these skilJs. As a result, we built 
task analy si s cycl es 1 nto_our Prof osslonaL Deyel ppment 
Center programs focusing on the low scoring skill areas 
Identified, by the State Assessment Program. 

The district's cadre of leaders began by training principals to ex- 
amine SAP (and later the other tests) to see what specific skills they 
assessed. Once these were identified, the jiext step was for principals and 
their faculties to examine school curricula 1n order to determine whether 
these skills were being taught and if so at whM; grades and with what em- 
phasis. Staff development provided principals, and later teachers, with 
the information and techniques they needed to do this. 

vThis was taking place with varying degrees of thoroughness 1n dif- 
ferent Bayview schools when CSE's Evaluation Design Project staff made Its 
several visits. At the same time, areas of curricular and instructional 
weakness district-wide had been Identified by district administrators. 
These areas were then targeted for. sessions on diagnostic-prescriptive 
teaching and other Instructional skills. 

, Analysis of test results also suggested areas for emphasis 1n the 

development of continua. Citing the Impact of proficiency-test skill and 

score analysis, for example, the Bayview Coordinator of Curriculum said: 

The proficiency *xam has helped the district focus on 
curriculum... [We learned, that] in math we teach computation. ; 
but the test tests applications through story problems. 
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Thus, 1n the Bay view Unified School District, task analysis of tested 
skills served as ti?e basis for a comprehensive examination of the dis- 
trict's curricula and suggested areas of curricular weakness. Simultane- 
ously, analysis of test* results led to the identification of teaching weak- 
nesses. Links between testing and instruction were generated through the 
development of d1str1ct-w1de objectives and 1n Professional Development 
Center and Teacher Center programs. * — — — 



Guidelines 

The St. John and Bayvlew districts had put in place very different 
kinds,-of systems for linking the results of externally mandated testing* 
with Instructional planning In their schools. Nevertheless, 1t 1s possible 
to abstract a number of guidelines from their "outside 1n" models. Other 
districts would be well advised to bear these 1n mind should they. follow 
similar approaches. 

1. Make test score data comprehensible for teachers and principals . 

Providing test results 1n a format that facilitates their use 1s obvi- 
ously a key to test1ng-1nstruct1 on linkage. That professional educators 
working 1n the schools can be, bewildered and Intimidated by reports of 
scores from externally mandated measures was clear 1n Test Use Project 
fleldwork -(cited early on 1n this paper). It was equally apparent 1n the 
early experiences of district administrators 1n both Bayvlew and St. John. 
The latter addressed this problem by translating the scerr- succinct, 
easy-to-read, and relevant tables and graphs. Bayvlew clea t c; with 1t by 
teaching principals and teachers to dissect the tests and test results. 

• :. ■ - : ' sci . ■ 156 ' ' ■ ■ : 
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2. Train teachersand principals to use test scores as diagnostic tools . 

As noted near the outset of this paper, the results of externally man- 
dated tests are commonly used In a brief £nd casual way to ge s t a general 
comparative reading on group performance. The essence of their use In the 
m St. John and Bayvlew systems was diagnostic^ They played a role 1n Identi- 
fying patterns of strength and weakness 1n particular .content areas and 
skills. They served to stimulate questions such as "Why are we scoring as 
we are scoring in this area?" and "How can we Improve?" Diagnostic uses 
are not routine 1n most schools. Simply presenting test scores 1n clear, 
readable format does not mean, that they will occur. Teachers need teaching 
and practice 1n hypothesizing the different factors that underlie test per- 
formance. They need Instruction and help in abstracting meaning from 
scores. Survey findings suggest that most districts do not provide this. 
In different ways, both St. John and Bayvlew did. 

3. Expect that results of externally mandated tests will serve as only one 
source of .information 1n planning and decision making . 

Wisely, neither Bayvlew's cadre of leaders nor St. John's district 
evaluation specialists tried to make test results the) sole basis for edu- 
cational decisions. Human values and priorities do and should Influence 
decisions about what objectives to pursue 1n school Improvement or to build 
Into district cohtlnua. The day-to-day experiences with students and ob- 
servations of their woFk that teachers and principals rely upon so heavily 
are hardly irrelevant in making instructional decisions. These factors 
were routinely accepted, along with test data, as bases for decision making 
by St. John administrators as they assisted School Planning Councils and 
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reviewed their plans. Bayview's Coordinator of Staff Development, too 
recognized that test data needed to be examined In light of other factors. 
"When we see through our task analysis and curriculum review what we are 
and are .not teaching, the next step is to ask, 'Do we or don't we want to 
teach this? How Important 1s 1t for our students. 1,1 

Data from externally mandated tests can serve to Identify problems, to 
support or dTsconf 1 r^exper1^ce~-ba sed ~j augments," and tq^stlmul^ate^ques- 
tlons. It can be used to justify or rationalize decisions that have al- 
ready been made. But as the separate experiences of St. John (recall their 
needs assessment questionnaires) and Bayview (recall their juxtaposition of 
multiple .measures to district curricula) Indicate, test data in themselves 
are only one Important source of Information for educational planning. 

Sunapary and Conclusions 

CSE's national survey and its fieldwork in two research projects 
suggest that both testing that is internal to the school and that which 1s 
externally mandated can be used more fully In systematic educational deci- 
sion making. Districts can build a curriculum and tests that can serve 
teachers 1 routine classroom needs and simultaneously provide consistent, 
reliable, and valid data for school and district decision making. Dis- 
tricts can also capitalize upon data from externally mandated testing by 
adapting 1t to local needs. No single approach or model will be appro- 
priate to every setting. But whether a district chooses to pursue linkage 
from the inside out or from the outside 1n, there are several factors that 
will be necessary for success. 
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One of these is district leadership. In each district studied by CSE, 
there was an individual or a small group in the district office ~ idea 
champions and supporters who were vitally interested in using test data 
in instructional planning and decision making. CSE's national test use 
survey substantiates that such leaders, make a difference 1n school -level 
uses of test Information. 

A second element 1n district success is an organizational arrangement 
— a setting and set of v procedures — for decision making. In Central City 
"schools there were the weekly meetings of unit teams; in St". "^hnT^gtil^r 
sessions of the School Planning Councils. Shelter Grove held Its princi- 
pal -teacher planning days 1n June, October, and February each year. In 
Bay view, the locus of linkage was staff development workshops, continuum- 
building committees, and regular school faculty meetings. These organiza- 
tional arrangements motivated arid structured the use of test results by 
creating (1) real needs for information, and (2; procedures by which the 
Implications of test-score patterns could be discussed and acted lipon. 
None of the field study districts with successful linkage systems simply 
offered schools test data and left their use to chance. 

Third, each of the districts managed testing and/or test results such 
that they Increased the marginal utility of test information for teachers 
and principals. Teachers routinely receive data on student achievement as 
they watch their students 1n class, review their assignments, and grade 
classroom tests. This data Is Immediate, rich and compelling. So too 1s 
the Information principals regularly gather as they talk with staff and 
visit their classrooms. To be as useful and as compelling, external test 
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Information must add "something new" to what teachers and principals 
already know. Each of the four models described above did this. Central 
City's computer-scoring-and-analysis system for unit tests summarized indi- 
vidual students 1 mastery of objectives, as well as their errors and weak- 
nesses. Shelter Grove compiled data on the progress of Individuals and 
instructional groupings toward benchmark goals. St. John's Academic Per- 
formance Profiles charted year-to-year trends and annual gains. Bayvlew's 
task analysis projects based on tested skills and test scores helped to 
reveal why and how students' performance was what 1t was. In each case, 
test datd was configured In ways that told teachers and principals 
something more than "your students are doing well 1n this and not so well 
in that" — which is Information teachers and principals typically feel 

they already have. v 

A fourth and final element in successful district linkage is the 
maintenance of on-going resource and support systems. In the districts 
studied, these centered 1n the area of staff development: training in test 
development and use, training 1n how to realize Instructional goals derived 
from test information, or both. Frequently, too, instructional support 
staff — learning specialists, media specialists, evaluation specialists — 
were routinely available to provide help and answer questions. Support 
also took the form of adaptability and flexibility on the part of district 
administrators. Clear channels were open for Central City and Shelter 
Grove teachers to participate in the development of and criticize the qua- 
lity of district curriculum and tests. St. John's evaluation specialists 
revised district needs-assessment surveys 1n light of teachers' feedback; 

. \ 
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lc;;al schools could add survey items suitable to their particular con- 
cerns. Bayview district leaders showed patience and understanding in 
encouraging principals and teachers to take a "problem-solving approach" to 
low test scores. And of course, each district supported its testing- 
instructional linkage system with release time and other' resources. 

The models and guidelines suggested here will not answer all the ques- 
tions and concerns school districts will encounter as they work to link 
testing and instruction in systematic ways. But they do indicate produc- 

'i ) \ 

tive directions to the more efficient u,se of testing and the improvement of 
education planning in American schools. 
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THE SEARCH FOR CONSEQUENCES: ASSESSING THE IMPACT 
OF DISTRICT INSTRUCTIONAL INFORMATION SYSTEMS 

a by 

Adrianne Bank and Richard C. Williams 

Background %y ' 

. Before we came along, Lenny, the Research Director in North District, 
and Don, the Testing and Special Education Director 1n South District', 
didn't know that their respective districts had given birth to Instruc- 
tional Information systems. Each knew only that his district had spent 
considerable time and attention on methods for combining student achieve- 
ment test scores with other district data so as to produce Information 
"useful for Instructional planning and management. Each knew that, for the 
past five or six years, he had worked with other key central office staff 
to provide Information, training and support to school -site groups and 
individuals. 

For our part, we at UCLA's Center for the Stud(y of Evaluation had, 
for several years, Investigated what school district research and evalua- 
tion offices did. (Lyon et al., 1978). He had discovered that most R&D 
personnel spent much of their' time administering d1 strict-wide testing 
programs and conducting mandated state or federal evaluations. We found 
that policy making, administrative decision making and classroom operations 
were rarely affected by the work done by the RSD unit. In re-anal yzlng our 
questionnaire data/ doing further Interviews and reviewing the literature 
on school districts as organizations, we discovered a number of reasons for 
this phenomenon iBank ft. Williams, 1981a). Many school districts 
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exist in a socially turbulent environment where critical factors such as 
annual budget levels, numbers of students*, placement of students in 
schools, and personnel matters relating to principals and teachers, are not 
exclusively 1n district hands. Outside forces areoften overwhelming. 
Furthermore, the Internal controls that°central office staff exert over 
school principals and classroom teachers are "loosely coupled" (Weick, 
1976). ?hat is, teachers are quite autonomous "behind their classroom 
doors" (Lortie, o 1975) . Schools each have their own community and culture; 
although districts differ, most have no strong guidelines as regards teach- 
ing methods, although most do,, have ^general guidelines for curriculum., 

*" a - ,0 - 

. ' - ( O 

Given these features, it is unusual for school districts: to plan and carry 
out systematic and centrally-directed activities relating to instruction.^ 
Because there is an absense of such centrally-coordinated instructional 
decision making, it 1s not surprising that centrally collected testing and 
evaluation data are regarded primarily as useful signals to, funding agen- 
des that the school district is complying with program evaluation require- 
ments (Zucker, 1981); t - 

Nonetheless, there are exceptions; there are districts where dis- 
otrict-directed testing and evaluation activities are linked With instruc- 
tlon. Lenny's district and Don's district are two of the eight "heroic" 
districts that we studied £ver the course of several years. In these dis- 
trlets, particular factors in the external and the internal .circumstances 
of the districts —.'including strong leadership, a critical mass of sup- > 
porters, a benign environmental setting — facilitated the central offices 
assuming a- role 1ti the Improvement of instruction (Bank and H1111amr., ^ 
1981b, 1982). In Lenny's and Don's districts, the district central office 
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provided relevant, timely and easily understood data to key decision makers 
(e.g., district administrators, principals and teachers). 

While we were conducting our case studies in Lenny 's and Don's dis- 
tricts, it occurred to us that there was a relationship between the infor- 
mation systems they were developing and the management information systems 
discussed in the business community. Our investigation into the management- 
information research literature revealed that there were indeed parallels. 
In our view, the parallels overshadowed the differences. And so we coined 
the construct "instructional information systems" {IIS) (Bank & Williams, 
1983). Instructional information systems (IIS) may not be pictured on an 
organization chart, but they can be described in terms of an observable set 
of components: specified users, specified uses, specified data inputs and 
outputs, a delivery system with particular format and timelines, and a 
monitoring and feedback mechanism. 

Once having concluded that such systems in either complete or incom- 
plete form do exist in districts, we set out to examine certain features of 
those systems. We used Lenny's and Don's districts as field sites. In 
another article (Catteralli 1983) we looked at the costs associated with 
Lenny's and Don's systems. In this paper, we will \e specifically con- 
cerned with the impact district instructional information systems have on 
the individuals they are intended to serve. 



Questions of Interest in Assessing impact 



Our interest in studying the impact of a district-operated IIS on 
system users is a natural one. Given the commonplaces in the educational 
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administration literature that, in loosely- coupled settings, policies man- 
dated from the top are considerably altered as they filter down to those 
expected to implement t hem, we wer e curious as to what might be the impact 
of Lenny's and Don's systems on teachers, principals and parents. 

Our major question, therefore* was "What is the impact of the instruc- 
tional information systems in these two districts?" 

Before thinking about what ^procedures we could use to answer this 
question, we had to answer several prior questions for ourselves. 
Question #1. How should the impact of an IIS be defined? 

Considerations : The ultimate expectation in both districts is that 
the,1nstructional Information system could Increase student 
- learning as measured by total, subgroup, or Individual changes on 

achievement test scores.; However, the causal linkage between 
information provided to adults, who then make well-informed " 
decisions which lead to Improved classroom practices which, 1n 
turn, lead to Increased student learning showing up as higher test 
scores was very weak. We decided that we could not Identify the 
impact of an Information system simply by tracking student test 
scores over time. 

Conclusions : Instead, we focused only on the Intended users of the 

^^^"^— 5. 

Instructional information system those adults to whom the dis- 
trict suppl ied information. We defined Impact as composed of the 
attitudes of Individuals and groups towards the information, and 
their behaviors 1n rel\t1on to the information, e.g., how they 



felt about it and what 



they did with 1t. 
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Question #2. How should impact of particular kinds of information supplied 
by the district be distinguished from other information used by individuals 
to make instruct!" onally-related decisions? 

Considerations : It has been pointed out that the single decision 
maker making a particular decision at a given poinl; in time using 
a limited set of data is an unrealistic view of how people in 
organizations operate (Alkin, et al ., 1979; Cronbach, et al.,, 
1980). How then could we separate out the influence; of the 
district aggregated and distributed information from the "working 
knowledge" (Kennecfy, 1982) that such people already possess? 
Cone! usion : We decided that our respondents themselves would be the 
best judges as to what influenced their thinking. We would ask 
them to discuss a decision that they had made using district- 
supplied information and estimate the importance of that infonna- ' 
tion in their own decision-making. We would also ask them about 
the particular contribution of each aspect of the IIS. 
Question #3. How should "ripple effects/ 1 often termed unanticipated, 
unintended or secondary impacts, of the instructional information system be 
identified and described? 

0 t Considerations : These impacts, we believed, would not be part of 
the "official" story that people told spontaneously. These 
o° effects would have to be inferred, carefully, by researchers using 
L ss interview data, cross-checked with observations. However, t we 

warited to capture the ripple effects because they would shed light 
o on the informal, daily life consequences of instructional infor- 
* mation systems. 

'. ■ " ' 167 
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Conclusion : We decided to include in our interview a list of possible 
"ripple effects" and ask respondents to. react With any examples 
that occurred to them. We also asked them to imagine what they 
and others would do if the district- supported IIS was disbanded, 
. either partially or completely. 

Methodology for the Study 

Issues . A number of factors influenced the way in which we approached 
the study of impact 1n the North District and the South District. 

First, we wanted to establish collaborative relationships with both 

Lenny and. Don, with whom we had previously worked, in order to make the 

study of use to them as well as to ourselves. Since the IIS had been 1n 

place 1n each district for a number of years and since 1t appeared to be 

operating smoothly and without major problems, neither Lenny nor Don was 

r 

motivated to assess Impact without our outside impetus. Each had his own 
concerns that the study might needlessly raise teacher doubts or anxieties 
about the system; or perhaps surface expectations for additions or Changes 
that they would not be able tc act upon because of other district priori- 
ties. Together with Lenny and Don, we discussed how to "position" the 
study 1n the eyes of the respondents so as to minimize the potential risks 
that Lenny and Don saw, and how to frame the Interview questions so that 
Don and Lenny too, might derive useful Information from our data. 

Second, we wanted to design a simple procedure that v:ould be appropri- 



ate for use\by other districts who desired to do their own impact studies. 
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The methodology therefore needed to be inexpensive, short-term ^nd accept-^ 
able^to participants. 

Sample , In each district we ^selected sample schools where implemen- 
tation was assured — that is, where "everyone knew" that individuals and 
groups were aware of and were using district-supplied information for 
decision making. In Lenny r s large district, m selected five elementary 
schools of varying sizes, and one high school. In Don's smaller district, 
we interviewed in all seven elementary schools. 

At eacn site, 8-10 individual teacher and parent respondents were 
selected by the principal of the school in accordance with our request for 
a variety of perspectives, experience and attitudes towards the IIS. 

Instruments . He considered and rejected questionnaires as Impractical 
because of the difficulties of 'getting a <vigh rate of return and because 
they would be more high-profile than both districts wished. Interview 
schedules with the same format were developed but they were Individualized 
for each school. Respondents were asked about their own background; then 
they were. asked to reconstruct a decision or a process which 1nvo1 ved t them 
with the district's IIS; they were asked to identify the importance of each 
system element in that process; finally, they were asked to react to a 
number of possible "ripples."* We then asked how they arid their group 
would react 1f the district discontinued the IIS; and finally, we asked 
them to add anything else they thought would be of interest to us. 

Analysis and Implications . Interview notes from each, district were 
transcribed and analyzed, site by site, to determine school variations. 
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With1n-d1str1ct themes across schools were also Identified. However, 
comparisons of the districts with one another were not appropriate as the 
two instructional information systems differ from one another in terms of 
intended purposes and _ operations. 



North District: Assessing the Impact of the District's 
Instructional Information System 

\ v ■ , 

\ 

PURPOSE OF THE SYSTEM 

It should be noted at the outset that while the North District's IIS 
was originally generated at the school district level and currently remains 
a central administrative responsibility, the purpose of the system- is to 



provide pi anning-relevant data and a planning and decision-making procedure 
for local school site councils, the decision-making responsibility resides 



at the local level — the district merely facilitates the local decision- 
making and planning proce^ It follows that questions about the impact o. 
district's instructional information system should focus upon whether or 
not this system does Indeed influence local school site councils 1n the way 
it was designed to do. \ 

The central office collects pupil^ achievement data and needs assess- 
ment data, processes those data and fashions them into sets of easy-to-read 
tables with accompanying narratives. These easily allow school site coun- 
cils to determine trends and identify strengths and weaknesses in the 
school's program. The d1str1ct\has adopted an Implicit, linear decision- 
making model that has several discrete stages. Data from the district 
^eem can presumably have, some ^Influence at\each stage Of the decision* 
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making or planning process. The decision-making stages and* the role of 
data can be briefly described as follows. 

1. Problem identification and clarification means determining if 
there-are shortcomings or problems at the school and once having identified 
such problems, initiating programs or activities that would be expected to; 
solve or reduce the problems. > H 

Of cpurse, many people involved in a school can perceive that there 
are problems in their school. Often such perceptions are based on a single 
experience or hearsay,. And often, based on ; such ill-informed perceptions, 
schools launcii reforms or changes. The goal of North District's instruc- 
tional information system 'is to make data-based decision making more syste- 
matic, comprehensive and objective. 

° Needs assessment survey data, collected from large numbers of 
parents, educators and students, can identify whether there are 
commonly perceived problems, the strengths of those perceptions and 
the degree to which they are cownonly held among the various con- 
stitutent groups. Such data reduces the likelihood that one influ- 
ential individual or group of individuals can overwhelm others with 
its version of what the problems are. 

° Similarly, norm-referenced test data can be used to identify 
possible weaknesses in the instructional program. If 3rd grade 
students 1 spelling scores, for example, are below a desired level, 
the reasons for this might be exploded. They could range from a 
spelling text-test* mismatch, to lack of pupil instruction, tQ 
inadequate amounts of instructional time; 

° Of even greater potential decision-making and planning use is the 
interrelationship between the needs assessment data and the test 
score data. That is, one could see if there were parallels between 
commonly held perceptions of the program and the test data. 

2. Solution generating and solution selection . Once a school site 
problem has been identified or clarified, one might next move to generaliz- 
ing solutions and then making selections among -the generated options. 



These data, particularly the needs assessment survey data, can be 
especially useful 1n this, regard. School site council members, empowered 
to develop solutions, can represent parent as well as teacher (and, 1n some 
cases, student) thinking. Informed voting should result 1n the selection 
of a useful solution. 

3. Program/activity evaluation and monitoring . Once a decision has 
been, made and a program or activity has been implemented, test data and 
opinion questionnaires can be very useful in determining whether or not the 
Intended outcomes were realized. , For example, picking up on the earlier 
Instance of low test scores in spell 1ng; r suppose one of the school site 
councils 1n North District had*dec1ded to provide a creative training pro- 
gram for teachers on how to effectively teach spelling. The Council might 
reasonably expect that such training would ultimately result 1n better 
pupil achievement, which would be reflected 1n the students 1 achievement on 
a standardized test. By observing the test scores for subsequent years, 1 
the school site council might have an information base to supplement 
teacher judgments and student homework to ascertain, the effects of its 
policy. Similarly, differences 1n the annual need's* assessment regarding 
the respondents 1 perceptions about some previously Identified problem could 
provide another Information base. 

In summary, the district should expect the Impact of Its Instructional 
information system to be felt at three /level s: problem Identification and 
classification; solution generation and selection; and program/activity 
evaluation and monitoring. 
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IMPACT O r 

We n,. ...>• findings with regard to primary and c -o ary im- 

pacts of the district's system; and we report the ripple and unexpected 
impacts of the system. 

The reader should note that we are looking at the impact this IIS 
has on the decision-making and program implementation activities of the 
school site council. We are not looking at the ultimate consequences of 
the decisions that were made. To illustrate — we are looking to see if a 
school site council 's decision to spend money on Inserylce training for 
teachers in, say, spelling instruction was influenced by data provided by 
the district's IIS. We are not looking to determine if the Inservice 
training of teachers ultimately has its desired effect namely improving 
the pupils' spelling ability. This latter — ultimately the most interest- 
ing and important question — is beyond the scope of this study. 

1. Problem identification and clarification . If the district's IIS 
Wfcl 'ng as designed and intendeds one would expect that the 

needs assessment survey of staff, parents, and pupils and the annual ITBS 
results would be used to identify those weaker programs'and activities in 
need of school site council attention and funding. 

In each of the six schools we visited, we asked the respondents to 
recall a major decision that their school site made during, the 1981-82 
academic year and. to reflect on how members had come to identify that area 
cr „uii<- .i. The impression that emerges from the six schools is that the 
test data played only a minor role as a means by which. the schools identi- 
fied the needed improvements. The survey data also played a minor role, 
being used to bolster an argument or proposal that had been advanced by a 
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faculty member or group. One school site council, for example, decided to 



Invest 1n a science program. This uec»s1on was t.. w <- o the arfmnnt of 
one teacher whb^ had a strong Interest In enhancing the science program. 
Three school s 1 site councils had decided to purchase computers for their 
schools. In all\three Instances they had been Influenced by a faculty 



member or .administrator who convinced the council that purchasing computers 

\ ■ ' 

would be a wise use of funds. In another school the school site council 

\ 

decided on a multl -cultural mathematics Instructional program. This pro- 
gram had been tried In other district schools with some success and 
teachers pushed hard,\and successfully, to have the program funded 1n their 

own school . \ 

Thus, from the sample six schools we saw little evidence that either 
the test data or the ne^ds survey data played a decisive role 1n the school 



site council's decision-making processes as regards the major items we 
discussed with them. \ 

While the respondents \d1d not Identify the Information supplied by the 
district as having a decisive role, they did Identify several Instances 
where the surveys and the test data alerted them to special needs that 
required additional funding. \The following are some examples: One elemen- 
tary school changed their math^books on the basis of ITBS test scores which 
Indicated lower pupil performance than expected. Another school uses a 



participative, non-h1erarch1cal decision-making procedure and administra- 
tive structure, wt\ere teachers and parents are regularly Involved 1n the 



school's Instructional decision-making. Here, the district's Information 
system has been thoroughly Integrated Into the school's dedsl on-making 
style. Everyone 1s so used, to this model that 1t is difficult for them to 
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see where their own system ends and their needs for district-supplied data 
beg^n. They cannot conceive of operating their school without either 
system, 

do .ot k t .,, c"H thf» Instances where the district-supplied data 
may have been used in identify ing-iw^ds. It Was cleu^ that in five of the 
six schools, school site committees were following the recommended district 
decision-making model and were looking at the survey and test data. 
Although the Impact of these data was not as clearcut as we had expected, 
it was obvious that people were famfilar with them and consulted them when 
making plans and decisions. v i\ 

One explanation for this 1s somewhat less-than-consensual use of the 
system 1s that this procedure is carried out annually and has now become an 
established way of Identifying problems over several consecutive years. 
Because the data from year to year do not change very much — that 1s, test 
scores do not fluctuate very much and parent and staff opinion probably 
remains qylte stable — the school site councils continue to fund programs 
they identified in previous years. And thus, when we asked about one year 
and about one decision, we did not surface dramatic uses of the data for 
problem Identification purposes. However, it 1s likely that data-based 
decision making has had a cumulative effect on the participants 1n the 
school site councils. They might be very concerned about making major 
decisions^ without a quick check of the data to see if they contained any 
strong contra-1nd1 cations or problems demanding more immediate attention. 

2. Solution generating and solution selection . If the district's In- 
structional Information system were working as designed, one would expect 
that the school site council , or more often, the component committees of 
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the school site council would comb the surveys and norm-referenced test 
data for their Implications as to solutions to the problems identified in 
the first part of this decision model. 

This did not turn out to be the case. With regard to the major deci- 
sions that we explored with our respondents from the six schools: their 
decisions were not based on needs spotlighted by the data nor were the 
solutions necessarily Implied by these data. For example, the decision to 
start a .Math-Physical Education flip-flop schedule was based on teachers 1 
awareness that this program had been used successfully 1n other schools 1n 
the district. The teachers were experiencing a problem 1n teaching math. 
They thought class sizes were too large. They knew that other schools had 
resolved this problem with the "flip-flop" and they persuaded the council 
to spend its money to Implement such a program.. 

. In the schools that decided on purchasing computers,, there appeared to 
be little searching for other options. In one school, the council wanted 
to "have the most computers of any school 1n the district." In part this 
desire was an answer to their perceived need to compete successfully with a 
nearby private school. . But data about parent preferences or student per- 
formance had little Influence on this decision. In another computer- 
purchasing school the decision was motivated by the fact that a group of 
rapid learning students were coming 1n to the school with their owrTclass- 
room computers. Largely 1n response to this, the school site council v 
^decided to Invest 1n computers for the remaining students. 

It is Important to note, however, that this school site council felt\^ 
the need for data to help with a related decision. There was a split 1n ^ 
the council as to whether the school should Invest in computers or air \ 

176 



169 



conditioners (the newly arrived rapid learners also had their own classroom 
**** conditioners). This dilemma was resolved when the school site council 
devised its own parent questionnaire on this specific question. The final 
decision choice was based on the results of that survey. Thus, while the 
results of, the district's survey were not used, the district's survey 
method was indeed used in resolving this* dilemma. Jhus, in a very real 
sense, it can be said that the district's surveys affected "the search for 
that solution. 

The decision to invest money in more teacher aides in another school 
was not influenced by data; indeed,, the needs survey data did not contain 
anything which would bear on that decision. The council went along with 
staff ideas about what was important to help the teachers in their work. 

With regard to the smaller decisions that are made by the school site 
councils 1 component committees, one senses that data are sometimes us6d to 
identify problem areas but that the selection of the solution does not 
arise from exploring the implications of the data. Instead the component 
committees select solutions based on common sense or on what other schools 
are doing. 

It appears in some schools that decisions of where to allocate funds 
are based in part on making sure that the various curricular components of 
the schobf site council- have their "fair" share of the money. Once an .77 
equitable division is decided- upon, the component committees determine how 
to spend the money.. The small amount of allocated funds does, of course, 
limit the range of solutions they can consider. 

Our research methods may well;* have limited our perceptions of the 
effect the district's management information system has had on the school 



site council 1 r decision-making processes . It may well be that what we 
perceived as a very, limited search for alternative solutions may Indeed be 
the residual from very extensive alternative-solution searches that were 
conducted in years past. And likely the limited amounts of funds now 
available to each component reduce the tendency for heroic, wide-ranging , 
flights into generating unconventional alternative solutions. 

3. Program/activity evaluation and monitoring . If the district's 
Instructional Information system is working as, designed and Intended, one 
would expect that the data from both norm-referenced tests and the surveys 
would be used by the school site councils to assess whether previous acti- 
vities and programs had achieved their desired effect. For example, 1f a 
component committee had funded the teacher In-service training program 1n 
spelling to Improve the quality of spelling teaching, one might expect that 
the students would perform better on subsequent spelling tests or that the 
parents 1 or teachers 1 . attitudes toward the need for spelling Improvement 
would change. Such Information probably should influence council decisions 
regarding the future continuation of the program. 

With regard to the major decisions we saw little evidence of the data 
being used that way. '■ The respondents had some firm Ideas about whether or 
not their council's programs had been Implemented and about their relative 
success ~ but the district-generated data didn't play much of a role in 
.determining those opinions. 

For example, one school had Implemented staff development programs. 
At the time we visited the school, our respondents reported that the In- 
service program had Indeed been Implemented but was not haying Its Intended 
Impact; there was considerable disappointment with the results. The 



respondents seemed to base their opinion on their own experience in the 
program and on conversations with colleagues who had also, participated, 
rather than on parent survey responses. 

In a school that had earlier adopted a Math-Physical hciucation and 
Multicultural "flip-flop* 1 they decided to drop the program even though many 
people felt it had been successful. The basis- for the decision was that 
the teacher who was responsible for the program became "burned out" and no 
longer could continue in that position. 

In some activities and programs, e.g., the computer purchase, there 
appears to be no felt need to have exhaustive data. For example, once^the 
computers are purchased .it is unlikely that that decision will be reversed 
in the forseeable future. Nor is it likely that the computers will have 
any measureable impact on test scores or parent and sU - : f attitudes for 
several years, if then. Again, the survey may eventually pick up future 
changes in knowledge and/or attitudes but the decision to actually buy a 
computer or additional computers will probably not be affected by such 
data. It may affect the priorities for use and types of computer programs 
employed, but it is too early to detect any trends along that line. 

With regard to the "smaller" decisions made by the school component 
committees it is likely that the test and scoring data are used to moni- 
tor programs 1 success more in some components than in others. Some respon- 
dents talked about how they would "set a target figure for an increase in 
satisfaction level" but no mention was made about whether or not much 
attention"was;given^ to whether or not the -target-was hit — and the subse- 
quent consequences of this relative accuracy. In that same school, a re- 
spondent stated, "We compare this year's figures to last year's figures. 
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If we staged the same because our scores were high we don't need to change 
anything, But if there was a drop in scores, we set a percentage goal, and 
we ask what it is that we have to do to get better scores, is what we are 
doing worthwhile?" 

Because of the large number of component committees in the several 
schools we visited, it is very difficult, to give anything more than general 
impress^ extent td which the data were being used to -monitor - 
the effects of decisions^previously made. Clearly, these data were being 
used in this way in some schools and in some component committees. Many 
respondents simply did not comment on this function and it is difficult to 
attach meaning to this- One reason for this might be, indeed, that the 
data simply are not used for this purpose but are used more in' originally ° 
deciding to spend money on a particular activity. It is likely that some 
decisions, such as hiring teacher aides, do not result in measureable 
differences and thus, these data are not very valuable in assessing 
impact. In other instances, it may be difficult to extract the" use of such 
data from the regular flow of deci si on-moni tori ng^revi sing-decisions that 
characterizes the functioning of some school site component committees. 

In summary, this monitoring question appears to be a regular part of 
the decision process in some schools and component committees. However it 
did not surface in revising and monitoring the large decisions such as 
computers; Math-PE-Mul ticul tural "flip-flop" and the science curriculum 
program. 
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OTHER IMPACTS 

In the preceding section we discussed how the district's instructional 
information system, i.e., providing school site councils with the test and 
survey data, has impacted on the decision-making process in the councils. 
Here, we turn to impacts that may not be directly related to a data-based 
decision-making process. The impacts we noted from the open-ended 
responses can be categorized into communication, parent participation, 
school culture and climate. 

1. Communication . Common sense would lead us to believe that an 
information system has high potential for improving or enhancing the level 
of communication in the school both within and among the various consti- 
tuent groups (parents, teachers, administrators, pupils). There are 
several ways it might have a practical effect. . - 

A. Provide a common language or common data base that can be tapped 
into by the various groups. A common complaint about educators is 
that they have their own professional language and culture and this 
makes it difficult for those who are not a part of that culture or 
who do not fully understand the professional language to communi- 
cate very effectively, (e.g., parents, pupils). The district's 
management information system has a powerful potential' for bridging 
this communications gap in that*all the participants have the same 
data — and those data are arranged in such a way that they are 
quite clearly understandable to the various constituent groups. 

B. A second, related effect is that it reduces the potential control 
jaf^those who have sole access.. to initial information. Thus, as is 

ofterkthe case with school site councils, the principal's or 

staff 'sXpotential ability to manipulate events is minimized because 



174 



. cr1t 1c.l Information Is .is. sv.11.Me to other decision-making 'or 

planning participants. 
■C It essentl.Tly forces various constituent groups to come together, 
to access a common data set and to reach consensus. In th,s pro r 
. cess, perceptions are; shared, Ideas are explored and decisions are 
reached. This results In bringing together people who night not 
otherwise work together In such an Intense way towards common 
goaU. The effect is that erroneous perceptions e.ch constituent 
group might have for the others ere reduced or corrected somewhat. 
D. Finally, It results In a communications network In that each member 
of the school sue council and Us various component committees has 
his . or her own groups of friends and acquaintances with whom to 
share information about the school's activities and their percep- 
tions of people and events In and around the school. This network 
can help to break down communication barriers and enhance the level 
of school -community relations, 
in visiting the schools, we ...-evidence of these activities In most 
schools; the level of influence the Instructional Information system had on 
each school varied depending on circumstances and motivations. 

in- the hloh schoo! it had considerable effect, particularly 1n bring- 
ing people together - people who typically do not meet to discuss, schoo, 
p la „s and activities. Parents' Interest In participating In school fan- 
ning typically falls off at the secondary level . But the schoo! slte-coun- 
c1! provided an avenue and means for including some parents wishing to be 
evolved in decision making. .S1.1l.Hy. thehigh school's typical division 
into. subject matter specialists and departments tends to reduce Inter- 
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faculty involvement in school-wide activities. The school site council 
tended to encourage such cross-departmental communications. 

in one relatively non-graded elementary school the district's instruc- 
tional information system served to enhance and support an ongoing, high 
level of faculty communication. The school's design and operations already 
encouraged considerable internal faculty communications - this district 
program empowered the already extant system to be even more effective by 
providing vast amounts of useful data. 

In other elementary schools it brought together parents and teachers 
into problem-solving groups that used the data to identify and attack prob, 
lems and to enrich the school -community relationship. _ ;■ 

But the existence of this system did not guarantee that this hap- 
pened, in one school the main purpose seemed to be a means whereby the 
faculty could purchase additional items and people to enhance their teach- 
ing. This, of course, is not an indesirable effect, but the program as 
practiced in this school virtually ignored the parents and minimized the 
teacher cooperative decision making. Instead, the faculty divided up the 
allocated money and bought what was viewed by each individual teacher as 

» » 

best for him or her. 

in summary, we saw the instructional information system having varying 

effects in the schools depending on circumstances. Overall, jt seemed to 
'..have a strong impact on communications but that impact was riot guaranteed 
' ~ individuals in the school had to value the data and believe in the 
system's decision-making and planning model . 

2. Parent participation . As has been noted in the previous section 
. on communications, the district's instructional information system brings 
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various constituent groups together (teachers, administrators, parents, 
pupils). Thus, parents are included as a part of the school's decision- ■ 
making body ~ at least as concerns the expenditure of SIP or Chapter 1 
funds. \ 

The school site council membership is mandated by law and is quite 
similar from district to district. But, as has been previously pointed \ 
out, being on a school site council should not be equated with part1cipa-\ 
ti on,' that is, often such councils are dominated by the professional \ 
staff. What is unique here is that the parents, thYough the systematic 
provision of data in a concise and understandable form, enhance the likeli- 
hood that parents can actually participate in the school site council's 
planning and decision-making process. 

The varying level of parent participation has been noted previously. 
Perhaps it is sufficient to say that these data, presented as they are 
along with a decision-making model, seem to have had the desired effect on 
increasing parent participation in those schools where the staff has an 
inclination to include parents. In our sample of six schools, we would 
conclude that parent participation was adequate in five of the schools. 
One note of interest: all the schools reported difficulties in maintaining 
the appropriate -level of parent participants largely because women, who are 
the main participants, are increasingly working during the day and 
unavailable to participate; , 

3. School culture/climate . The district's instructional information 
system might reasonably be expected to impact on the decision-making 

processes and 1t is not surprising that it influenced the communications 

* 

pattern and interactions and affected parent involvement. Here we discuss. 
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that we really hadn't anticipated, namely the culture of the school. It. is 
this to which we now turn. 

Typically, ^school teachers, administrators and parents are not known 

\ * 

for utilizing data to identify problems, determine solutions and implement 

I 

plans or decisions. Indeed, the more common portrait of teachers is that 
of working alone behind their closed classroom doors. Principals are often 
portrayed as being more preoccupied with administrative matters than with 
instruction and programs. Staff meetings more often deal with management 
and budgetary matters than with instruction or program items. These 
program, curricular and instructional changes or developments that are made 
are seldom influenced by 3 test data or any comprehensive set of survey 
data. Teachers spend their time alone in the classroom; principals busy - 
themselves- with administrative and management concerns. Program- 
matic-planning is minimal and generally uninformed by data collection. 
Parent involvement is often limited to fund raising, and sponsoring and 
running various school activities through the PTA or at the classroom level 
through "class mothers." Clearly, the district instructional information 
system has changed this standard culture in the schools we visited. Some 
of these changes have been noted, e.g., communications patterns, parent 

o 

involvement, decision making. But another important change or impact was 
also observed, one that is not quite included in these activities. For 
lack of a better term, we will call it a spirit of inquiry — orientation 
toward problem solving — a bias towards data. 

We saw this impact time and agairj, as we observed school site councils 
at work and as we gathered the perceptions of those who participated in 
these activities. The component conmittees seemed to constantly refer ta 
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data when reaching decisions. This is not to say that the data provided by 
the instructional information system was always critical to the decision. 

To be sure, other factors such as personal opinions, professional judgments 

> 

and strong advocates often prevailed. Sometimes, the data merely 
corroborated what the group wanted to do anyway. But collecting and 
analyzing°data had clearly become something the participants valued and 
indeed were comfortable with. 

Perhaps this approach can. best be illustrated by the events in two 

schools. In the sample high school, they were wondering about whether or 
not to support a staff development program in the school. Faced with 
uncertainty over whether there was support for such a program, the staff 
devised its own survey questionnaire which was distributed to all faculty 
members. (The district questionnaires did not adequately cover this 
topic.) The resulting data was analyzed and the program decision was made 
on the basis of. those data., 

In one of the elementary schools, the school site council was faced 
with a choice between buying air conditioners (for year-round classrooms) 
or purchasing a school computer. The results of the district question- 
naires did not give insights into such a specific question. The school 
site council devised its own questionnaire and, on the basis of the results 
from staff and parents, decided to purchase the computer. Other examples^ 
abound. One elementary school has devised its own- student survey^which is 
distributed regularly and the results are considered when^maKI ng funding s 
decisions. Another school has used the data when ^rilfing proposals for 

c 

funding from private sources and state agencies, 

V 
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- Not all of the data are usgd equally, however. We found very few 
respondents who referred to the ITBS scores when making decisions. There 
may be several reasons for this: the district's test scores tend to be 
uniformly quite high and, thus, the scores do not attract the attention 
they would if they revealed glaring deficiencies. Also, the pattern of 
test scores probably dpes not shift dramatically from year to year. Given 
the eveneness, one would expect that there would be little likelihood that 
the scores would underscore major deficiencies. Finally, the test scores 

"fit into a pattern a--stream--of--data — -~-an"d-^ 

test scores figured more prominently in decisions when the program was 
begun in each school. Because we didn't see them used now,^t does not 
follow that they were not used at an earlier point in the program* s 
history. The, scoring data was used far more extensively, but even here 
some data received more attention than others. The parent survey data 
received the most attention largely, we suspect, because it frequently was 
the only systematically collected, comprehensive data set of parent 
opinions regarding the school. There is no other way available to collect 
such dat£. These data, appeared to be examined quite carefully by the 
school site council. But the data were not without shortcomings. For 
example, the percentage of sample returns was l sometimes quite lev/ below 
50 percent. Also, some teachers felt there were val idity" problems and 
sometimes the parents were responding about things of which they had Httl 
knowledge. For example, in one school the parents said that the home * 
.economics program needecFimprovement. Subsequently, the home economics 
program was completely dropped from the school. 'In the next parent survey 
the. parents noted that the home economics 'program had improved! Even 

187 



180 

v 

though the data are not infallible, they do represent an Important data 
source and they receive a good deal of attention. 

The teacher surveys of data do not seem to receive as much attention. 
The main reason appears to be that the teachers have many opportunities to 
talk and discuss matters and' to influence each other. Thus, for the staff, 
the teacher surveys do not provide many insights or surprises. Probably, - 
the teacher responses may provide some insights to the parent members of 
the school site council's and component committees but parents, do not play 
as critical a role in the deliberations as does the professional staff. 

In summary, we were impressed that these, school s had a proclivity 
towards data and they valued a decision-making procedure that was really 
quitCi^pical of what would be-found in most schools. The principals and 
staff were not able to (nor were they inclined to) dominate discussions and 
decisions. Parents and, where appropriate, students, had a real sense of 
efficacy and participation. The wall that sometimes separates the public 
from its schools had been broken down and a bridge of communication and 
mutual trust- had been constructed in its place. 

South District: Assessing the Impafct of the District's 
In structional Information System 

This study is of a small district which has, over -the past eight 
years, created a workable system by which teachers continally tailor their 
instructional practices to the learning outcomes of their students. The 
district has created a centrally-mandated, school -managed, classroom- 
operated set of procedures that ensure a quality education for children. 
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„ h at is rif about South'* system Is not the Idea Itself. After 
personnel 1. most districts wouid say th.t quality teaching and learn- 
tne primary business of the public schools. Also not unique are the 
Mvldrt components of the system Testing, standard setting, text 
section, staff development, district-wide curricula, schoo, resource 
rooms, are co-on events in «t district settings. And what is unique 
about the system is not the .any hard-working teachers, satisfied parents, _ 
„gh achieving students. These, too, can be found in man, school 
districts. - 

pervasive, and self-renewing. A common orientation towards teaching and 
Uarning has become the customary way of life for students, teachers, 
principals, central off led personnel . board members, parents and others in 
the community. 

It was not always this way. In the late 60'.. the school reform 
splr ,t in the country proved South' s central office to action. The years 
of trial and error in developing a testing system, the early contentious- 
ness of school teachers and administrators about the curriculum, the ini- 
tial expenditures of non-productive dollars have now'become part of the 
district's folklore. Stories of these early days and of the personal toll 
they exacted form a background against wbich to understand present-day 
proud counts from teachers sue* as. "we're all overachievers." "I went 
TO own children to go here." . "He can catch Incoming Oildren up to grade 
level within two years." "The puaHty of our schools would go down without 
the district system. . He are consistent from school to school ." 
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The following account looks at the impact of South 1 s instructional 
information system after a brief description. 
DESCRIPTION OF SYSTEM 

1. The criterion-referenced testing system . These tests, initially 
developed and continually revised by teams of teachers, are the major de- 
vices regulating instruction. The CRT tests, each taking no more than half 
an hour to administer, are given in each classroom three times a year, or 
more often at the individual teacher's discretion. Normally, teachers 
teach a unit of material in math, reading or language arts, then wait two 
weeks and give the test to their students. Test booklets may be scored by 
the teachers or may be machine-scored. In either case, the testing coor- 
dinator returns to the teachers' computer printouts with their students' 
scores organized by objectives and printed out by learning group. Print- 
outs are also available by child, by class and by school. 

2. The district continuum . All CRT tests are referenced to milestone 
objectives on a kindergarten-through-eighth grade instructional continuum 
for reading, language arts and math. The tests provide a gross diagnosis 
of student progress through each subject sequence. After teachers review 
the test' results, they, decide whether to move their instruction back or 
forward along the continuum. Each teacher uses the tests for a rough 
assessment of individual performance, that is, for projecting and measuring 
student growth on the continuum over the course of a school year. All 
teachers as a group work with the school principal to Identify grade level 
•Instructional priorities according to student performance on the tests 
since the results allow teachers and administrators to determine the 
percentage of students performing below, at, and above grade level in a 
given subject area 9^ ; 
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The district scope and sequence for reading, language arts, and math 
is a graded set of objectives accompanied by suggested teaching methods. 
The continua contain more objectives than there are CRT tests. They are 
sequenced and constitute a minimum set of expectancies for children by 
grade level. The continua are filed in looseleaf books, well-thumbed and 
referred to continuously by all teachers. 

3, The Professional Development Program , The Professional Develop- 
ment Program (PDP) provides teachers with the skills necessary to act upon 
the results of the CRT tests. The PDP is coordinated by a full-time 
specialist who either teaches all the courses or hires consultants to do 
so. In addition, some PDP courses are taught by district teachers. The 
program has* evolved over time. 

The PDP program, as well as teaching particular instructional tech- 
niques to all teachers, has developed for teachers and principals a common 
vocabulary in which to discuss children's learning. Clinical supervision 
of teachers by principals is made easier by a (common frame of reference and 
a common understanding about desirable teaching strategies. Teachers and 
learning specialists, both of whom have gone through the same PDP program, 
have a set of methods for acting upon the results of the CRT tests. 

4. Learning specialist . The learning specialist might in other 
school- districts be called the vice principal. In South the learning spe- 
cialist does not have either the administrative or the disciplinary func-. 
tions generally associated with the vice principal role.* The learning 
specialist is a master teacher who functions as a resource to teachers and 
makes it possible for teachers to act upon the diagnoses .implied by their 
children's responses on CRT tests. The learning specialist brainstorms 
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with the teacher about instructional alternatives. More imporw^t, he or 
she provides extra Instructional time, on a pull-out basis, for children 

who need it. - 

The learning specialist coordinates and facilitates the CRT testing, * 
making sure that the tests are given, that the results come back on time, 
and that teachers review and act upon them. Additionally, the learning 
specialist provides, for new teachers and for new policies, in-service 
sessions about the functioning of the CRT system. 

5. School principal . Principals are expected by the district to , 
spend a good deal of time in the classroom. Principals regularly walk 
around the. halls and drop into classes in addition to doing their formal 
classroom observations. The formal observation may either be invited, 1f 
the teacher is nervous or new, or requests it, or uninvited if the princi- 
pal feels comfortable about doing it. The principal visits have an evalua- 
tive function: "I am the clout that backs up the learning specialist" says 
one principal. "I evaluate, th3 learning specialist helps." 

Additionally, the principal, during the end-of-year planning days, 
meets with the teachers and discusses the expected progress of groups of 
students for the year. During the course of the school year, principals 
receive student scores and look them over for progress and "surprises. 11 
Teachers are held accountable for student performance — not for having all 
children achieve all of the preset goals — but for explaining deviations 
from' teachers 1 earlier expectations. 

Principals 1n South are themselves evaluated both informally and 
formally in conjunction with the amount of time they spend on teacher 
supervision and checking on students 1 progress. 
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Principals meet regularly with the superintendent. Discussion of 

- I 

student progress is a regular, periodic part of their agenda. 

PURPOSE OF THE SYSTEM 

All districts, when asked to describe the important features of their 
instructional programs will likely mention the goal of improving pupil 
achievement and they will insist that a major district practice is to 
individualize instruction to the needs of students who have differing 
skills and capacities. Most often, however, the attainment of this 
individualized instruction goal is left to the devices of individual 
teachers who work with minimal supervision behind their closed classroom 
doors. Some teachers are quite skilled at individualizing instruction ~ 
some are not. While school districts might offer some supervision and 
occasional in-service training in how best to individualize instruction, 
the degree to which such individualization occurs is largely incidental to 
any school district effort or intervention. 

Not so in South District. While South might resemble many other 
districts in regard to the expressed goal of improving pupil achievement 
through individualized instruction, it differs considerably in the extent 
to which the district actively involves itself in assuring that this goal 
is realized throughout the district. Teachers in- fc each South District 
classroom are expected to individualize instruction as a result of a 
comprehensive instructional program that is directed by the district 
central administration, but that reflects the teachers 1 concerns and 
Intents. 

If asked to select one word to define the essence of this program, we 
would choose integration. The entire system is designed* to integrate the 
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elements listed earlier, i.e., criterion-referenced testing system, dis- 
trict continuum, professional development program, learning specialists, 
and principals, into a tightly-coupled system that assures that .teachers 
are" skilled at individualizing instruction and are aware of the progress 
their pupils are making towards mastering the components of the school 
district's instructional continuum. 

The school district, in integrating these five elements into a common 
program, provides the teachers with tools and concepts they can use in 
their classrooms. For example, they have a series of teacher-developed, 
district criterion-referenced tests the teachers canine to diagnose pupil 
achievement and, when appropriate, 'to prescribe remediation or change the 
instructional pace or method. Also the teachers are given intensive in- 
service training to provide them with the skills to meet individual pupils' 
varying instructional needs. Also the teachers and principals, and indeed 
the parents,- are provided with a set of terms and methods that translate 
into a common language about the instructional program and the pupils' 
progress through the program. This reduces considerably the mi scommuni ca- 
tion and misunderstandings that are often found 4 n many school districts.. 

Keeping the above in mind," the impact of this system should be mea- 
sured by the extent to which classroom teachers consider that the system i 
useful U> them in meeting the overall goal of enhancing pupil achievement 
through individualizing instruction. 
IMPACT OF THE SYSTEM 

When we began our impact inquiries in South District, we entered a 
system that had been developing and in operation for many years. From 
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previous visits we knew that the reading and mathematics parts of the pro- 
gram had been operating for many years. While we could ask "impact ques- 
tions" about these two instructional fields we were unsure if our teacher 
respondents would be able to differentiate between the district instruc- 
tional information system's impact and the extent to which current prac- 
tices contained some unknown carryover from practices that predated the 
district's instructional information system. 

In an effort to get as clear a picture as possible of how the instruc- 
tional information system worked and. its impact, we decided to select a 
recent addition to the system and measure its impact. We chose the 
district's new writing instruction program as the focus of our inquiry. 

In a nutshell, the district had become aware that some parents and 
teachers were concerned that the schools were not. adequately teaching the 
students to write creatively, clearly or accurately. After collecting 
criterion-referenced test data and visiting other districts and writing 
projects (one principal visited some British schools), the district decided 
to launch a comprehensive program to improve student writing throughout all 
grades. This decision was reached after considerable discussion and 
analysis which included principals, teachers and parents. 

All of the system components were brought to bear on this project. 
Teachers helped develop a special criterion-referenced testing series; 
teachers also helped develop a district continuum of K-8 writing skills, , 
which was built upon the Bay Area Writing Project; almost all the teachers 
have attended a voluntary inservice program that instructed them in the 
fundamentals of teaching writing, of diagnosing and treating student 
writing problems and motivating students to want to write creatively and 
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well. Learning specialists helped teachers with diagnostic problems and 
worked with small groups of students with common problems; principals 
visited classes often to assess the teachers' progress and to /make program- 
matic adjustments, where warranted. 

One of the first questions we explored was the extent to which this 
program had been implemented. In a typical loosely-coupled school dis- 
trict it is not uncommon for the central administration or board to mandate 
a particular program and find, after some period of time, that the program 
had not been implemented as designed, or that it had been differentially 
implemented, that is, implemented in some schools or classrooms and not in 
others. 

So, a first question we explored before^trying to measure impact was 
the universality of implementation. After completing our field work, we 
were convinced that the- district's writing program had been Implemented in' 
every elementary school in the district and in almost all the classrooms. 
" There were some few teachers who did net participate in the program but 
their number was small . One can say with considerable confidence that the 
program has been implemented; parents can be quite sure, that their children 
enrolled in the South School District will experience a comprehensive and 
sequenced instructional program in writing. 

The'Wa^iers and admi^strators wfth^^Tini»e-taTked^^t1f1ed a large 
number of impacts.- For the purpose of brevity we have reduced their com- 
ments into three categories: academic learning time, school climate and 
culture, and communications. 

Academic learning time . . Derived from the Beginning Teacher Evaluation 
Studies, this term means that pupils spend an adequate time on an 
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instructional task at a level that allows the student to achieve mastery 
and to.be challenged to move, when appropiate, to higher levels of achieve- 
ment. Once we sorted through all the teacher's comments, this theme 
emerged clearly. It was manifested in several ways, such as: 

0 The start-up time at the beginning of the year is greatly reduced 
because the teacher has criterion-referenced test results on each 
of the new pupils on each of the district continuum elements. 
After a brief check to see if the test results are accurate the 
teacher can immedately begin to work with students, at each 
student's appropriate level. . 

0 The criterion-referenced tests, 'which can be administered and 
corrected in a short period of time, provide teachers with what 
they generally view as a valid measure of pupil progress. They can 
quickly see where the gaps are in pupil learning, and they can 
place students into temporary instructional groups with peers who 
have similar instructional needs. 

0 The teachers can be quite specific when talking to- parents about 
the students' instructional needs and the kinds of home activities 
that will help students achieve mastery of the subject matter. 

0 The negative effect of having substitute teachers is diminished in 
this program. Substitute teachers can be provided with a rather 
specific individualized plan to. follow with classes. 

School climate and culture . Typically in elementary schools teachers 
work quite independently behind their classroom doors. To be sure they 
follow a general district pattern of studies, that is, a third grade 
teacher attempts to cover' "3rd "grade materi aU" But the- teachers are 
loosely supervised by the principal; the principal generally does not • 
interfere with a teacher' s activities unless it is plainly clear that the 
teacher i s qu i te Incompetent. Itls difficult for the p7fficTpST^rH>the.r 
teachers to determine the extent to which individual teachers are meeting 
the instructional needs of Jfidivi dual pupils. One could, thus, character- 
He the culture of most/scfibolis as minimizing organizational cooperation. 
There 1s little sense "of teamwork ; the school does not work as an -* 
integrated whole. , 
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In South District schools, one senses a much different culture and 
climate. With regard to the teaching of- writing, the climate can be best 
described as teamwork — as a group of teachers and a principal and learn- 
ing specialist working toward a common goal in a common way. All the 
teachers teach writing in the same way and a teacher at one grade, say 4th 
grade, can be sure that the students have had a similar writing experience 
with their 3rd grade teacher. What is more, the fourth grade teacher can 
use CRT's in identifying specific knowledges or skills that each student 
may not have mastered in the previous year. ; 

"The relationship between the principal and teachers 1s also quite dif- 
ferent. Principals visit classrooms regularly and have a common under- 
standing with the teachers about what should be occurring in the class- , 
rooms. But the atmosphere or mood during these visits is not one of super- 
visor "checking up" on subordinates, but is Instead one of a colleague who 
is interested in another's work. Principals do not report being concerned 
about teacher competency; the teachers have been carefully selected and 
trained by the school system, and they receive* continuous feedback on their 
effectiveness. 

What is more, the teachers have been, and continue to be involved in 
developing and refining the system in use. Thus, there is none of the kind 
of "sabotaging 11 that sometimes occurs when new instructional materials are 
implemented from the top without teachers 1 involvement. 

Communications . Finally, an important impact of this system is the 
degree to which it eases communication among the various groups that make 
up a school, i.e., teachers, principals, supervisors, and pare nts. As h as 
been previously noted, this system provides a common set of expectations, 
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methods and outcome measure. It also provides a common vocabulary. The 
effect of this on communications among teachers, principals and learning 
specialists is obvious: they can quickly understand each other and prob- 
lems are easily placed in a familiar cohtext. 

One of the greatest impacts this common set of expectations and voca- 
bulary has seems to be on. the communications between teachers and parents. 
Instead of the more common teacher-parent conference where the parent is 
informed about the pupil 's progress in terms of how he or she compares with 
other students or the level of effort the student is putting forth, South 
District teacher-parent conferences can focus on those skills and know- 
ledges from the district continuum. that the pupil has mastered or not 
mastered. At the beginning of each school year, the teacher and parent 
review the child's CRT results and a specific teaching, and study strategy 
is mapped out for the coming year. The roles of the teacher, parent and 
student in achieving the desired goals are agreed upon and clearly under- 
stood. At subsequent conferences during the year the student's progress is 
noted and adjustments are made, if heeded. 
OTHER IMPACTS 

" ■ Because this^ is a carefully designed, comprehensive system that has 
been fully implemented, there were few unexpected impacts. Those that were 
initially encountered, such as teacher resistance, have been ameliorated. 

■The only impacts-that-appeared to be of any consequence were: time commit- 
ments, computer. and measurement errors, and an inordinate number of intra- 
di strict transfers into South District* 

Time commitments . Numerous teachers ^commented on. the amount.. o.t time. 

the CRT testing program took in their classes. However, they also noted 
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that the district no longer gave NRT's and thus the total time devoted to 
testing was perhaps not significantly greater than that which is spent on 
testing in other districts. The teachers who commented on. the time commit- 
ment, however, also felt that the time was well spent because it resulted 
in data that helped them to check pupil progress and individualize 
instruction. 

Computer and measurement error s. Many teachers observed that they 
must read the computer data very carefully because occasionally the compu- 
ter results are erroneous. Also, the CRT's are not infallible. Students 
occasionally score considerably above or below their real level of compe- 
tency. Usually, when this occurs, the teachers catch the error and either 
retest or use their own judgment in planning the student's instructional 
program. In short, this system, with all its sophistications, cannot 
totally replace teacher judgment in making instructional decisions. 

Intra-di strict transfers . The state 1n which South District is 
located allows parents from one district to apply to have their children 
transferred into another district. Because of South District's reputation 
'and its instructional program, the district believes that it has had an 
inordinate number. of parents from surrounding districts who wish to have 
their children transferred into South District. This does not represent a 
problem — it is merely an unexpected impact. 



Summary 



Having reviewed the impact of, these two quite different instructional 
information systems, we will conclude with some final observations about 
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the similarities and differences between the two and what this tells us 
about measuring impact. 

Differences, 'Clearly, these two systems are aimed at serving quite 
different audiences. North seeks to inform school site councils; South, 
the classroom teacher. North's system is far more indirect in its imme- 
diate impact on the classroom. Indeed, much of the impact may not have any 
direct measurable effect on individual classrooms and teachers. The two 
systems, quite expectedly, differ in the level and consistency of imple- 
mentation. North District portrays considerable variation in the way and 
degree to which its system has been implemented. South District, on the 
other hand, is quite uniform and comprehensive in implementation from 
school to school. Finally, the two districts face different problems as 
regards the further implementation and development of their programs. 
North must address reasons for the differential implementation from school 
to school and seek to understand the reasons for this variation and, if 
necessary, consider strategies for overcoming undesirable variations«in 
program implementation. 

South, on the other hand, must assure that their program does not 
become a victim of its own success. That is, success sometimes breeds 
complacency which then-results in the organization losing its drive and 
capacity to approach new problems with a fresh perspective. 

Clearly, if a district is going to try to measure the impact of its 
Instructional information system, it must make sure that it has a clear 
understanding of what _the system is intended to accomplish, the degree to 
which the system components have been developed, and the conditions that 
allow the district to make reasonable progress. 
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Center for the Study of Evaluation 
Evaluation Systems Project 
Interviews/ /Principals/Learning Specialists fc 

We are interested in your views about the district's testing, curricu- 
lum amd instructional system, how it functions and the impact of its opera- 
tion on teaching. By this system, we mean ■ 

Let's take as an example, the teaching of writing {or math, reading) 
during this past year, in the lower grades. , . 

1. Can you describe how writing is supposed to be taught; in this m * • 
district? 

2. What percentage of teachers do it this way?. What accounts for those 
who do and do not teach that way? . ; ' ' . 

3. In your view, what influence jdoes each of the following components 
have on the teaching- of writing? . 

the district's scope and sequence , . . ■, 

CRT tfests and feedback , 
learning specialist 

professional development program • 

media resources " 

teacher evaluation procedures • 

4. Do these answers apply to instruction in math? in « reading?^ ^ 

5. What difference would 7 it make if the district 'no longer required the 
use of the scope and sequence, CRT's and pDP? First, tell me your 

• own reaction. 'Then tell me what effect this would have on the 
schools in general. , ■*; 



6. (Next page) 
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Center for the Study of Evaluation 
Evaluation Systems Project 
Interviews/ /Teachers 

We are interested in your views about the District's testing, curricu- 
lum amd instructional system, how it functions.'. 'and. its Impact on teaching. 
By this system, we mean the tests and the district's scope and sequence. 

Let's talk about your approach to instruction. Let's take writing (or 
math or reading). . 

1. Could you tell me how writing is supposed to be taught in this 
district? 

2. Do you teach it in this way? Do most teachers? (Probe: what 
accounts for those who do and do not teach that way?) 

3. How are you influenced in your teaching of writing by . » 

the district's scope and sequence? 
CRT tests and feedback? 
learning specialist? 
professional development program? 
media resources? 
teacher evaluation procedures? 

4. "Do these answers apply to the way you teach math and reading? 

5. What difference would it make if the district no longer required the 
use of the scope and sequence, CRT's and PDP? First, tell me your 
own reaction. Then tell me what effect this would have on the 
schools in general. 



6. (Next page) 
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I'm going to go through some impacts that the system might have on different groups 

of people. In your school /has the CRT/curriculum/PDP affected 

IMPACT 



teachers 1 

a. coGiaini cation 
with one another?^ 

b. coainuni cation 
with parents'! 



c. understanding of 
test scores? 



d. use of test data in 
in making decisions 
about change^ 



e. problem solving skills^ 
x. other 



on principals 

f . awareness of student 
achievement 



g. corrmini cation 
with staff 

h. comnunication 
with parents - 



x. other 



on parents 
i. attitude toward district- 

j. knowledge of school 
programs 



k. attitude toward schpol 
x. other 



on students 
1. leading progress 



m. attitude toward school 



n. relationships with peers _ — ; — ^ 



o. relationships with teachers : — — ^ 

p. relationships with parents 
q. feelings of accomplishment 
x* other . 



. Name School ^OS Position^ 
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Center for the Study of Evaluation 
Evaluation Systems* Project 
Interviews/ 

We are interested in your views about your school's School Site 
Council, how it functions, and the impact of its decisions on the school 
and community, 

' I understand you were on the SSC last year, I would like to discuss 
both how the SSC functioned and to get your assessment of the impact the 
SSC's decisions have had on the school this year. 

*\ 

1. I understand the SSC decided last year to 

How did you come to be working in that area? 

What evidence did you have that led you to consider this as 
an area for SSC action? 

What was the sequence of events that led to the SSC's final 
decision? 

test data used? survey data used? process used? 
committee consideration? subcommittee ;work? 



2. What happened as a 1 result of that decision by the SSC? 

How did people become aware of the decision? , 
What did people do about the decision? Why? 
How do you know? 

What have been positive, and negative impacts of this decision? 

3. Is this typical in its process? In its impact? Explain. 

4. Generally, of what use to the SSC are a) the test data, b) tjpe survey 
data, c) the district decision-making process, d) district training? 

5. Suppose that the district decided to discontinue collecting survey data 
and giving you test data. What would your reaction be? 

6. What difference do you think- it would make if the district's system 
supporting the School Site Council were discontinued? 

7. tSee next page) ■ ' 



*with principals, SSC teacher and parent partldants 
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7. I f m going to read a list of impacts that a SSC might have on different groups of 
people. In relation to your own school situation, how would you characterize each? 

IMPACT 



On teachers 

a. communication 
with one another 

b. communication 
with parents 

c. feelings of efficacy 

d. understanding of 

test scores _ 

e. understanding of 

cormunity attitudes 

f . use of data in 

making decisions 

g. problem solving skills 

x. other 

On principals 

i. management skills ! 

j. budgeting skills 

k. sense of fiscal 

accountability ; 

1 . awareness of student 

achievement 

m. comnuni cation 

with staff 

n. conmini cation 

with parents _ 

x. other . — - . — 

On parents 
o. awareness of schools" problems 
& efforts to deal with them 

p. knowledge of school 

programs , , 

g. a t titude toward school 

x. other, ; 



None 



Positive 



a little 
1 



a lot 

3 



Negative 



a little 
1 2 



a lot 

3 



Don't 
Know 



Name 
Role 



School 
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Interviewer " Date __ __ Time 

Name of Respondent School 

Rol e • Length of time in SSC 

" o 

I 

[interviewer's Record of Responses] 
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